Repository navigation
fix(coderd): enforce chatd concurrent-agent capacity on every path into running - #30243
Conversation
…artbeats Three paths let more running chats hold fresh leases than the pool capacity allows: 1. Admit returned true without counting for any status other than running. A chat interrupted while it waited for capacity, or a requires_action chat whose replica died, was acquired without a slot, and FinishInterruption, Interrupt, CompleteRequiresAction or the action deadline then returned it to running under that ownership. Admission now applies to every runnable status, which matches the count query: an owned chat with a fresh lease already counts as holding a slot. A refusal now skips the rest of the pool's candidates in the pass regardless of status. 2. The heartbeat loop upserted every runner's lease unconditionally, so a replica that lost the database past the stale threshold revived a lease whose slot capacity admission had already given to another chat. BatchUpsertChatHeartbeats is replaced by RenewChatHeartbeats, which only extends rows that are still fresh and still own their chat. heartbeatOnce runs it under the capacity admission lock, using statement_timestamp() read after the lock, and requests cleanup of every runner whose lease was not renewed, so the stale runner stops generating. The chat stays owned by the stale lease until an admitted worker takes it over. When the pool is full, an unowned interrupting or requires_action chat now waits for a free slot before its runner starts; chats that already own a slot are unaffected. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
…at lock waits Follow-ups to the capacity admission fix: 1. Stopping a chat that waited for capacity hung in interrupting until a slot freed, because an interrupting chat now needs admission before a runner can finish the interruption. When Interrupt, or SendMessage with busy_behavior=interrupt, runs on a running chat with no owner, there is no runner, episode or tool execution in flight, so the transition now applies FinishInterruption in the same transaction, cancelling orphaned tool calls. An empty queue lands in waiting; otherwise the queue head is promoted into running, still unowned, so it goes through admission. Owned chats still land in interrupting. 2. Heartbeat renewal runs under the global capacity admission lock, so a stalled lock holder could block every replica's renewal past the stale threshold and then get every runner cleaned up. Each renewal tick is now bounded by HeartbeatRenewalTimeout (a third of the stale threshold) and its lock wait by HeartbeatLockTimeout through a new SetTransactionLockTimeout query. A failed tick cleans up no runners and retries on the next tick. Admission waits at most 5 seconds for the lock, so a stalled holder fails that candidate instead of the pass. 3. CountChatCapacityQueuedByPool now also counts interrupting and requires_action chats without a fresh lease, which wait for admission too. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
|
@codex review |
|
@codex security review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6bc935e2cb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A takeover reads the candidate's lease before it takes the capacity admission lock. A heartbeat renewal that holds the lock at that moment can still renew the lease, after which the takeover replaced a runner whose lease had just been renewed, and the old runner kept generating until pubsub or its next heartbeat tick stopped it. The takeover now rechecks the lease while holding the admission lock and skips the chat if the lease is fresh again. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
…al in ARCHITECTURE.md Describe admission for every runnable state, the takeover recheck under the admission lock, fenced heartbeat renewal with its timeouts and runner cleanup, and the inline finish of an interrupt on a chat that no worker owns. Replace the heartbeat upsert query with the renewal query. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1d71c73ace
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…h pool Admission now applies to interrupting and requires-action chats, so a saturated pool refuses them and they stay at the front of the candidate list. Ranking statuses globally let enough refused rows from one pool fill the limit and hide the other pool's lower-status chats. Rank by status inside each pool and interleave the pools by that rank instead. Also read ownership through the transaction store in the transitions matrix test helper.
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
|
@ibetitsmike thanks for the approval on
Codex code review and security review are both clean on |
Problem
The concurrent-agent cap (5 root chats and 10 subagent chats per deployment by default) could be exceeded. Three paths put more chats in
runningwith a fresh lease than the pool allows:busy_behavior=interruptto a chat that is waiting for capacity. The chat becomesinterruptingwhile unowned, a worker acquires it without admission, andFinishInterruptionpromotes the queued message into a running turn under that ownership.requires_actionchat. A replica dies while its chat is inrequires_action, its lease goes stale and another chat takes the slot. A worker acquires therequires_actionchat without admission, and a tool result, an interrupt or the action deadline returns it torunningunder that ownership.Root cause
agentCapacityLimiter.Admitreturnedtruewithout taking the admission lock or counting for every status other thanrunning. The count query already treats every owned chat with a fresh lease as holding a slot, so the shortcut let chats acquire a slot uncounted and later move back torunningunder that ownership. Separately,BatchUpsertChatHeartbeatshad no freshness or ownership condition.Fix
Admitnow counts forrunning,interruptingandrequires_action, and a refused pool skips the rest of its candidates in the pass regardless of status. An owned chat holds its slot until it releases ownership, so every later transition back torunningstays within capacity.BatchUpsertChatHeartbeatsis replaced byRenewChatHeartbeats, which only extends rows that are still fresh and still own their chat, usingstatement_timestamp()read after the capacity admission lock is granted. The renewal runs under that lock with alock_timeout(HeartbeatLockTimeout) and a per-tick timeout (HeartbeatRenewalTimeout, a third of the stale threshold). Admission also waits at most 5 seconds for the lock.heartbeatOncerequests cleanup of every runner whose lease was not renewed, which cancels its generation. The chat stays owned by the stale lease until an admitted worker takes it over. A tick that fails or times out cleans up no runners and retries on the next tick.interruptingwould have made Stop on a chat waiting for capacity hang until a slot frees. WhenInterrupt, orSendMessagewithbusy_behavior=interrupt, runs on a running chat with no owner, the transition now appliesFinishInterruptionin the same transaction and cancels orphaned tool calls. An empty queue lands inwaiting; otherwise the queue head is promoted intorunning, still unowned, so it goes through admission.CountChatCapacityQueuedByPool(behindcoderd_chatd_agents_queued_for_capacity) now also countsinterruptingandrequires_actionchats without a fresh lease, since they also wait for admission.How it was found
A TLA+ model of capacity admission found a counterexample for each path. Go tests reproduced all three against the previous code (each failed with
"2" is not less than or equal to "1"for a root pool of capacity 1), and the model with the fix enabled passes the safety invariant and the liveness properties. The model itself is not part of this PR.Tests
The three reproductions are kept as regression tests in
coderd/x/chatd/agentcapacity_regression_internal_test.go:TestAdmission_InterruptWithMessageOnWaitingChatStaysWithinCapacity,TestAdmission_RequiresActionTakeoverStaysWithinCapacityandTestRunnerManager_HeartbeatDoesNotReviveStaleLease. The original reproductions failed on the base commit. Restoring the admission shortcut makes therequires_actiontest fail, and restoring the unconditional heartbeat upsert, or dropping the runner cleanup, makes the heartbeat test fail. New coverage also includesTestAdmission_StopWaitingChatNeedsNoCapacity,TestRenewChatHeartbeats_OnlyRenewsFreshOwnedLeases,TestRunnerManager_FailedHeartbeatTickKeepsRunners, and chatstate tests for the unowned interrupt paths. Existing tests that encoded the old behavior (requires_actionchats holding no slot, interrupting chats always acquirable, unconditional heartbeat renewal) were updated.Behavior / API change
POST /api/v2/chats/{chat}/messageswithbusy_behavior=interrupton a chat that no worker owns (for example one waiting for capacity) now returns the promoted message withqueued=falseinstead of a queued message, and the chat stays inrunninginstead ofinterrupting. If the chat already had queued messages, the queue head is promoted and the new message stays queued.POST /api/v2/chats/{chat}/interrupton such a chat lands it inwaitingimmediately (or inrunningwith the queue head promoted) instead ofinterrupting.interruptingorrequires_actionchat (for example after its replica died) now waits for a free slot before its runner starts, which also delays enforcing its action deadline while the pool is full.HeartbeatRenewalTimeoutandHeartbeatLockTimeout, defaulting to a third of the stale threshold and half of that.Out of scope / follow-ups
worker_idset) still waits for a slot.queued_for_capacityflag in the chat API still reports onlyrunningchats.coderd/x/chatd/ARCHITECTURE.mdgains TODO lines for the affected sections; the prose is left for the owner.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high