Skip to content

fix(coderd): enforce chatd concurrent-agent capacity on every path into running - #30243

Merged
ThomasK33 merged 5 commits into
mainfrom
fix-chatd-capacity-bypasses
Oct 1, 2026
Merged

ThomasK33 merged 5 commits into
mainfrom
fix-chatd-capacity-bypasses

Conversation

@ThomasK33

Copy link
Copy Markdown
Member

Problem

The concurrent-agent cap (5 root chats and 10 subagent chats per deployment by default) could be exceeded. Three paths put more chats in running with a fresh lease than the pool allows:

  1. Interrupting a waiting chat. A user sends a message with busy_behavior=interrupt to a chat that is waiting for capacity. The chat becomes interrupting while unowned, a worker acquires it without admission, and FinishInterruption promotes the queued message into a running turn under that ownership.
  2. Taking over an orphaned requires_action chat. A replica dies while its chat is in requires_action, its lease goes stale and another chat takes the slot. A worker acquires the requires_action chat without admission, and a tool result, an interrupt or the action deadline returns it to running under that ownership.
  3. Heartbeat reviving a stale lease. A replica loses the database for longer than the 30 second stale threshold and another replica admits a second chat into the slot. When the first replica comes back, its heartbeat upsert renews the stale lease unconditionally, and its runner keeps generating until the turn ends.

Root cause

agentCapacityLimiter.Admit returned true without taking the admission lock or counting for every status other than running. The count query already treats every owned chat with a fresh lease as holding a slot, so the shortcut let chats acquire a slot uncounted and later move back to running under that ownership. Separately, BatchUpsertChatHeartbeats had no freshness or ownership condition.

Fix

  • Admission for every runnable status. Admit now counts for running, interrupting and requires_action, and a refused pool skips the rest of its candidates in the pass regardless of status. An owned chat holds its slot until it releases ownership, so every later transition back to running stays within capacity.
  • Fenced heartbeat renewal. BatchUpsertChatHeartbeats is replaced by RenewChatHeartbeats, which only extends rows that are still fresh and still own their chat, using statement_timestamp() read after the capacity admission lock is granted. The renewal runs under that lock with a lock_timeout (HeartbeatLockTimeout) and a per-tick timeout (HeartbeatRenewalTimeout, a third of the stale threshold). Admission also waits at most 5 seconds for the lock.
  • Stop runners that lost their lease. heartbeatOnce requests cleanup of every runner whose lease was not renewed, which cancels its generation. The chat stays owned by the stale lease until an admitted worker takes it over. A tick that fails or times out cleans up no runners and retries on the next tick.
  • Instant Stop on a queued chat. Requiring admission for interrupting would have made Stop on a chat waiting for capacity hang until a slot frees. When Interrupt, or SendMessage with busy_behavior=interrupt, runs on a running chat with no owner, the transition now applies FinishInterruption in the same transaction and cancels orphaned tool calls. An empty queue lands in waiting; otherwise the queue head is promoted into running, still unowned, so it goes through admission.
  • Queued metric. CountChatCapacityQueuedByPool (behind coderd_chatd_agents_queued_for_capacity) now also counts interrupting and requires_action chats without a fresh lease, since they also wait for admission.

How it was found

A TLA+ model of capacity admission found a counterexample for each path. Go tests reproduced all three against the previous code (each failed with "2" is not less than or equal to "1" for a root pool of capacity 1), and the model with the fix enabled passes the safety invariant and the liveness properties. The model itself is not part of this PR.

Tests

The three reproductions are kept as regression tests in coderd/x/chatd/agentcapacity_regression_internal_test.go: TestAdmission_InterruptWithMessageOnWaitingChatStaysWithinCapacity, TestAdmission_RequiresActionTakeoverStaysWithinCapacity and TestRunnerManager_HeartbeatDoesNotReviveStaleLease. The original reproductions failed on the base commit. Restoring the admission shortcut makes the requires_action test fail, and restoring the unconditional heartbeat upsert, or dropping the runner cleanup, makes the heartbeat test fail. New coverage also includes TestAdmission_StopWaitingChatNeedsNoCapacity, TestRenewChatHeartbeats_OnlyRenewsFreshOwnedLeases, TestRunnerManager_FailedHeartbeatTickKeepsRunners, and chatstate tests for the unowned interrupt paths. Existing tests that encoded the old behavior (requires_action chats holding no slot, interrupting chats always acquirable, unconditional heartbeat renewal) were updated.

Behavior / API change

  • POST /api/v2/chats/{chat}/messages with busy_behavior=interrupt on a chat that no worker owns (for example one waiting for capacity) now returns the promoted message with queued=false instead of a queued message, and the chat stays in running instead of interrupting. If the chat already had queued messages, the queue head is promoted and the new message stays queued.
  • POST /api/v2/chats/{chat}/interrupt on such a chat lands it in waiting immediately (or in running with the queue head promoted) instead of interrupting.
  • An unowned interrupting or requires_action chat (for example after its replica died) now waits for a free slot before its runner starts, which also delays enforcing its action deadline while the pool is full.
  • New internal chat worker settings HeartbeatRenewalTimeout and HeartbeatLockTimeout, defaulting to a third of the stale threshold and half of that.
  • Heartbeat ticks now take the capacity admission advisory lock briefly on each replica.

Out of scope / follow-ups

  • Interrupting a chat still owned by a dead runner (stale lease, worker_id set) still waits for a slot.
  • An admission lock timeout should end the acquisition pass instead of logging a warning per candidate.
  • The per-chat queued_for_capacity flag in the chat API still reports only running chats.
  • Alert on failed heartbeat ticks.

coderd/x/chatd/ARCHITECTURE.md gains TODO lines for the affected sections; the prose is left for the owner.


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

…artbeats

Three paths let more running chats hold fresh leases than the pool
capacity allows:

1. Admit returned true without counting for any status other than
   running. A chat interrupted while it waited for capacity, or a
   requires_action chat whose replica died, was acquired without a slot,
   and FinishInterruption, Interrupt, CompleteRequiresAction or the
   action deadline then returned it to running under that ownership.
   Admission now applies to every runnable status, which matches the
   count query: an owned chat with a fresh lease already counts as
   holding a slot. A refusal now skips the rest of the pool's candidates
   in the pass regardless of status.

2. The heartbeat loop upserted every runner's lease unconditionally, so
   a replica that lost the database past the stale threshold revived a
   lease whose slot capacity admission had already given to another
   chat. BatchUpsertChatHeartbeats is replaced by RenewChatHeartbeats,
   which only extends rows that are still fresh and still own their
   chat. heartbeatOnce runs it under the capacity admission lock, using
   statement_timestamp() read after the lock, and requests cleanup of
   every runner whose lease was not renewed, so the stale runner stops
   generating. The chat stays owned by the stale lease until an admitted
   worker takes it over.

When the pool is full, an unowned interrupting or requires_action chat
now waits for a free slot before its runner starts; chats that already
own a slot are unaffected.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
…at lock waits

Follow-ups to the capacity admission fix:

1. Stopping a chat that waited for capacity hung in interrupting until a
   slot freed, because an interrupting chat now needs admission before a
   runner can finish the interruption. When Interrupt, or SendMessage
   with busy_behavior=interrupt, runs on a running chat with no owner,
   there is no runner, episode or tool execution in flight, so the
   transition now applies FinishInterruption in the same transaction,
   cancelling orphaned tool calls. An empty queue lands in waiting;
   otherwise the queue head is promoted into running, still unowned, so
   it goes through admission. Owned chats still land in interrupting.

2. Heartbeat renewal runs under the global capacity admission lock, so a
   stalled lock holder could block every replica's renewal past the stale
   threshold and then get every runner cleaned up. Each renewal tick is
   now bounded by HeartbeatRenewalTimeout (a third of the stale threshold)
   and its lock wait by HeartbeatLockTimeout through a new
   SetTransactionLockTimeout query. A failed tick cleans up no runners and
   retries on the next tick. Admission waits at most 5 seconds for the
   lock, so a stalled holder fails that candidate instead of the pass.

3. CountChatCapacityQueuedByPool now also counts interrupting and
   requires_action chats without a fresh lease, which wait for admission
   too.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-01T12:14:44.011023Z 9306180 Manual request
🔒 Security Review ✅ Completed 2026-10-01T12:16:12.376715Z 9306180 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6bc935e2cb

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6bc935e2cb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/database/queries/chats.sql Resolved
Comment thread coderd/x/chatd/ARCHITECTURE.md Resolved
Comment thread coderd/x/chatd/ARCHITECTURE.md Resolved
Comment thread coderd/x/chatd/ARCHITECTURE.md Resolved
A takeover reads the candidate's lease before it takes the capacity
admission lock. A heartbeat renewal that holds the lock at that moment
can still renew the lease, after which the takeover replaced a runner
whose lease had just been renewed, and the old runner kept generating
until pubsub or its next heartbeat tick stopped it. The takeover now
rechecks the lease while holding the admission lock and skips the chat
if the lease is fresh again.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
…al in ARCHITECTURE.md

Describe admission for every runnable state, the takeover recheck under
the admission lock, fenced heartbeat renewal with its timeouts and
runner cleanup, and the inline finish of an interrupt on a chat that no
worker owns. Replace the heartbeat upsert query with the renewal query.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 1d71c73ace

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1d71c73ace

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/worker.go Resolved
…h pool

Admission now applies to interrupting and requires-action chats, so a
saturated pool refuses them and they stay at the front of the candidate
list. Ranking statuses globally let enough refused rows from one pool fill
the limit and hide the other pool's lower-status chats. Rank by status
inside each pool and interleave the pools by that rank instead.

Also read ownership through the transaction store in the transitions
matrix test helper.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 9306180582

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 9306180582

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33

Copy link
Copy Markdown
Member Author

@ibetitsmike thanks for the approval on 1d71c73ace. After it, I pushed 9306180582 to fix two Codex P1 findings. Could you take another look at that delta? @mtojek, an approval from you also works.

  1. GetChatWorkerAcquisitionCandidates now ranks candidates by status within each capacity pool and interleaves the pools by that rank. Before, status was ranked across both pools, so a saturated pool's refused interrupting or requires-action rows could fill the whole limit and hide the other pool's chats. This PR introduced that starvation when it made every acquisition go through admission. TestWorker_AcquisitionCandidatesRankStatusWithinPool fails on the old query and passes on the new one.
  2. The matrix test helper ownRunningChat now reads ownership through tx.Store() instead of a second connection.

Codex code review and security review are both clean on 9306180582.

@ThomasK33
ThomasK33 requested a review from ibetitsmike October 1, 2026 12:18
@ThomasK33
ThomasK33 merged commit d11809f into main Oct 1, 2026
49 of 51 checks passed
@ThomasK33
ThomasK33 deleted the fix-chatd-capacity-bypasses branch October 1, 2026 14:42
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants