Skip to content

perf(coderd/x/chatd): make chat stream reads lock-free - #30266

Merged
cstyan merged 2 commits into
mainfrom
callum/chatd-03-lock-free-reads
Oct 5, 2026
Merged

cstyan merged 2 commits into
mainfrom
callum/chatd-03-lock-free-reads

Conversation

@cstyan

@cstyan cstyan commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

ChatMachine.ReadLock took a FOR SHARE row lock (GetChatByIDForShare) to give multi-statement readers a consistent snapshot. The stream loop runs it on every snapshot_version bump for every subscriber, so each read serialized against the FOR UPDATE writers. The readers now run in a read-only REPEATABLE READ transaction (ChatMachine.ReadSnapshot): snapshot isolation gives the same multi-statement consistency without a row lock, and ordering is still handled by the snapshot_version watermark (an older snapshot triggers a later refetch rather than an inconsistent read). The lock statement also doubled as an existence check before the callback ran; nothing needs it, since every caller reads the chat as its first statement and already handles sql.ErrNoRows, so ReadSnapshot just opens the transaction and runs the callback.

Why this is safe: REPEATABLE READ gives the callback the same one-consistent-state guarantee the FOR SHARE lock did, and a read-only transaction cannot hit serialization failures; the only difference is that a transition committing mid-read is not waited for, which every caller already tolerates because the task starters and the hook server re-validate under FOR UPDATE before writing (fence and state checks) and the stream loop only applies snapshots by increasing snapshot_version. A transition that commits during a read is never lost: its chat:update hint is published only after commit, so the stream loop's next sync (driven by that hint, or by the 10 s streamSyncPoller if the hint is dropped) is guaranteed to observe it. TestStreamSyncObservesTransitionCommittedDuringRead exercises exactly that interleaving against a real database: a stream sync's snapshot is held open, a FinishTurn commits without waiting for it, the sync applies the pre-commit state, and the transition's own hint drives the next sync, which delivers the new status (the poller row would trigger the same fetch). TestReadSnapshotNotBlockedByRowLock covers the other direction, a read completing while a FOR UPDATE lock is held. Both fail against the FOR SHARE implementation. The task callers' re-validation is covered by the existing stale-fence tests (TestInterruptTask_StaleFenceExits and friends) and the hint-after-commit ordering by TestUpdatePublishesAfterCommit.

Database impact: in the scaletest runs this read showed up as heavy lock wait under load; readers now neither block nor are blocked by transitions. Wait sampling also had GetChatByID as the single largest database consumer, dominated by execution time rather than lock wait, and every stream sync paid for the chats_expanded read twice (the lock statement and the callback's own read); it now pays once.

Combined with the other PRs: none semantically; it touches only the read path, and the merge with #30267 is automatic (adjacent lines in machine.go).

Series: #30264 harness jitter, #30265 transition metrics, #30266 lock-free stream reads, #30267 single-write transitions, #30268 commit-write result and RBAC cache, #30269 non-blocking acquisition, #30270 finish turn in the step commit.


Generated by Coder Agents on behalf of @cstyan.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we have any existing tests that ensure the safety guarantees we're promising hold in practice? Should we have any?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not that I know of. This branches history is a bit off now, the agent decided to make some changes and then rebase/force push without asking, but the 2nd commit contains what feels like a reasonable test.

@ethanndickson

Copy link
Copy Markdown
Member

@codex review

@ethanndickson

Copy link
Copy Markdown
Member

Pinging Hugo on this since the chat state machine is his baby

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-02T07:29:20.672900Z d583cae Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d583cae276

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

) (dynamicPostToolUseState, error) {
var state dynamicPostToolUseState
err := machine.ReadLock(ctx, func(store database.Store) error {
err := machine.ReadSnapshot(func(store database.Store) error {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retain the row lock before dispatching tool-result hooks

When one SubmitToolResults request already holds the chat's FOR UPDATE lock and another request enters here, this snapshot can read the previously committed requires_action state instead of waiting. The second request then dispatches EventPostToolUse hooks in SubmitToolResults, but its later CompleteRequiresAction fails after the first request commits, causing external hook side effects for tool results that were never accepted. The former ReadLock waited for the writer and rejected this request before hook dispatch, so this pre-hook validation still needs locking even if stream-only reads become lock-free.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this is a pre-existing issue on main?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the root cause is preexisting, and this PR adds a new way for the issue to manifest itself. We should add a follow-up issue to investigate this.

@hugodutka hugodutka left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable.

@cstyan
cstyan force-pushed the callum/chatd-03-lock-free-reads branch from d583cae to e363352 Compare October 2, 2026 23:09
ChatMachine.ReadLock took a FOR SHARE row lock (GetChatByIDForShare) to
give multi-statement readers a consistent snapshot. The stream loop runs
this on every snapshot_version bump for every subscriber, so the read
serialized against the FOR UPDATE writers and showed up as heavy lock
wait under load.

Run the readers in a read-only REPEATABLE READ transaction instead, as
ChatMachine.ReadSnapshot. Snapshot isolation gives the same
multi-statement consistency without a row lock, so reads no longer block
or get blocked by writers. Ordering is still handled by the
snapshot_version watermark, so an older snapshot triggers a later
refetch rather than an inconsistent read.

The lock statement also doubled as an existence check that returned
ErrChatNotFound before the callback ran. Nothing needs it: every caller
reads the chat as its first statement and already handles sql.ErrNoRows
(the task loaders map it to errTaskExpectedExit, the HTTP handler maps
it to 404), so ReadSnapshot opens the transaction and runs the callback,
saving a round trip per stream sync.

Two concurrency tests pin the guarantees: an open ReadSnapshot neither
blocks a concurrent Update nor observes its commit (and the next read
does), and a ReadSnapshot completes while another transaction holds the
row's FOR UPDATE lock. Both fail against the FOR SHARE implementation.
Exercise the interleaving the lock-free read makes possible against a
real database: a stream sync's snapshot is held open after it reads the
chat, a FinishTurn commits without waiting for it, the sync applies the
pre-commit state, and the transition's own chat:update hint drives the
next sync, which delivers the new status. The poller row for the chat
would trigger the same fetch. The test fails against the former FOR
SHARE read because the transition blocks behind the held reader.

The chatstate-level test that asserted the same three properties in
isolation is removed as redundant; the reader-not-blocked direction keeps
its own test.
@cstyan
cstyan merged commit 1a96da2 into main Oct 5, 2026
74 of 77 checks passed
@cstyan
cstyan deleted the callum/chatd-03-lock-free-reads branch October 5, 2026 18:47
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 5, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants