Repository navigation
perf(coderd/x/chatd): make chat stream reads lock-free - #30266
Conversation
cafa79d to
d583cae
Compare
There was a problem hiding this comment.
Do we have any existing tests that ensure the safety guarantees we're promising hold in practice? Should we have any?
There was a problem hiding this comment.
Not that I know of. This branches history is a bit off now, the agent decided to make some changes and then rebase/force push without asking, but the 2nd commit contains what feels like a reasonable test.
|
@codex review |
|
Pinging Hugo on this since the chat state machine is his baby |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d583cae276
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| ) (dynamicPostToolUseState, error) { | ||
| var state dynamicPostToolUseState | ||
| err := machine.ReadLock(ctx, func(store database.Store) error { | ||
| err := machine.ReadSnapshot(func(store database.Store) error { |
There was a problem hiding this comment.
Retain the row lock before dispatching tool-result hooks
When one SubmitToolResults request already holds the chat's FOR UPDATE lock and another request enters here, this snapshot can read the previously committed requires_action state instead of waiting. The second request then dispatches EventPostToolUse hooks in SubmitToolResults, but its later CompleteRequiresAction fails after the first request commits, causing external hook side effects for tool results that were never accepted. The former ReadLock waited for the writer and rejected this request before hook dispatch, so this pre-hook validation still needs locking even if stream-only reads become lock-free.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Looks like this is a pre-existing issue on main?
There was a problem hiding this comment.
I think the root cause is preexisting, and this PR adds a new way for the issue to manifest itself. We should add a follow-up issue to investigate this.
d583cae to
e363352
Compare
ChatMachine.ReadLock took a FOR SHARE row lock (GetChatByIDForShare) to give multi-statement readers a consistent snapshot. The stream loop runs this on every snapshot_version bump for every subscriber, so the read serialized against the FOR UPDATE writers and showed up as heavy lock wait under load. Run the readers in a read-only REPEATABLE READ transaction instead, as ChatMachine.ReadSnapshot. Snapshot isolation gives the same multi-statement consistency without a row lock, so reads no longer block or get blocked by writers. Ordering is still handled by the snapshot_version watermark, so an older snapshot triggers a later refetch rather than an inconsistent read. The lock statement also doubled as an existence check that returned ErrChatNotFound before the callback ran. Nothing needs it: every caller reads the chat as its first statement and already handles sql.ErrNoRows (the task loaders map it to errTaskExpectedExit, the HTTP handler maps it to 404), so ReadSnapshot opens the transaction and runs the callback, saving a round trip per stream sync. Two concurrency tests pin the guarantees: an open ReadSnapshot neither blocks a concurrent Update nor observes its commit (and the next read does), and a ReadSnapshot completes while another transaction holds the row's FOR UPDATE lock. Both fail against the FOR SHARE implementation.
e363352 to
fe8cc3b
Compare
Exercise the interleaving the lock-free read makes possible against a real database: a stream sync's snapshot is held open after it reads the chat, a FinishTurn commits without waiting for it, the sync applies the pre-commit state, and the transition's own chat:update hint drives the next sync, which delivers the new status. The poller row for the chat would trigger the same fetch. The test fails against the former FOR SHARE read because the transition blocks behind the held reader. The chatstate-level test that asserted the same three properties in isolation is removed as redundant; the reader-not-blocked direction keeps its own test.
ChatMachine.ReadLocktook aFOR SHARErow lock (GetChatByIDForShare) to give multi-statement readers a consistent snapshot. The stream loop runs it on everysnapshot_versionbump for every subscriber, so each read serialized against theFOR UPDATEwriters. The readers now run in a read-onlyREPEATABLE READtransaction (ChatMachine.ReadSnapshot): snapshot isolation gives the same multi-statement consistency without a row lock, and ordering is still handled by thesnapshot_versionwatermark (an older snapshot triggers a later refetch rather than an inconsistent read). The lock statement also doubled as an existence check before the callback ran; nothing needs it, since every caller reads the chat as its first statement and already handlessql.ErrNoRows, soReadSnapshotjust opens the transaction and runs the callback.Why this is safe:
REPEATABLE READgives the callback the same one-consistent-state guarantee theFOR SHARElock did, and a read-only transaction cannot hit serialization failures; the only difference is that a transition committing mid-read is not waited for, which every caller already tolerates because the task starters and the hook server re-validate underFOR UPDATEbefore writing (fence and state checks) and the stream loop only applies snapshots by increasingsnapshot_version. A transition that commits during a read is never lost: itschat:updatehint is published only after commit, so the stream loop's next sync (driven by that hint, or by the 10 sstreamSyncPollerif the hint is dropped) is guaranteed to observe it.TestStreamSyncObservesTransitionCommittedDuringReadexercises exactly that interleaving against a real database: a stream sync's snapshot is held open, aFinishTurncommits without waiting for it, the sync applies the pre-commit state, and the transition's own hint drives the next sync, which delivers the new status (the poller row would trigger the same fetch).TestReadSnapshotNotBlockedByRowLockcovers the other direction, a read completing while aFOR UPDATElock is held. Both fail against theFOR SHAREimplementation. The task callers' re-validation is covered by the existing stale-fence tests (TestInterruptTask_StaleFenceExitsand friends) and the hint-after-commit ordering byTestUpdatePublishesAfterCommit.Database impact: in the scaletest runs this read showed up as heavy lock wait under load; readers now neither block nor are blocked by transitions. Wait sampling also had
GetChatByIDas the single largest database consumer, dominated by execution time rather than lock wait, and every stream sync paid for thechats_expandedread twice (the lock statement and the callback's own read); it now pays once.Combined with the other PRs: none semantically; it touches only the read path, and the merge with #30267 is automatic (adjacent lines in
machine.go).Series: #30264 harness jitter, #30265 transition metrics, #30266 lock-free stream reads, #30267 single-write transitions, #30268 commit-write result and RBAC cache, #30269 non-blocking acquisition, #30270 finish turn in the step commit.
Generated by Coder Agents on behalf of @cstyan.