Repository navigation
Conversation
Postgres re-runs every foreign key check on an UPDATE whose old row version was written by the current transaction. A transition used to bump snapshot_version first and then update the chats row again from the message history trigger, UpdateChatExecutionState, the attempt increment, the retry state write, and one queue-version trigger per queued row. Each follow-up write re-checked owner_id, organization_id, and last_model_config_id with FOR KEY SHARE on parent rows shared by every chat, so all transitions serialized on the users, organizations, and chat_model_configs heap pages (pg_buffercache showed every backend queued on those three pages). Each transition now ends with exactly one chats UPDATE that also advances snapshot_version: UpdateChatExecutionState, IncrementChatGenerationAttempt, UpdateChatRetryState, or the new BumpChatSnapshotVersion, which Update also issues when a callback performed no commit write. LockChatForTransition replaces LockChatAndBumpSnapshotVersion and writes nothing. The revision trigger stamps snapshot_version + 1 (the version the transaction commits); the history and queue-version triggers are dropped and the commit write records those changes from flags set by the message and queue helpers. InsertChat starts a chat at version 0 so CreateChat still lands on 1/1. UpdateChatExecutionState keeps clearing retry_state whenever the chat leaves running. The lock also changes from FOR UPDATE to FOR NO KEY UPDATE (as do GetChatByIDForUpdate, LockChatByID, and the context-resource sync): transitions never change chats.id, and FOR UPDATE conflicted with the FOR KEY SHARE locks that foreign-key child writes such as the batched heartbeat upserts take on the chat row, so heartbeats and transitions convoyed against each other. The lock's RETURNING row, extended with has_queued, seeds the Tx. The first loadState in a transaction returns it with no round trip. This is not a cache across writes: every transition validates before it writes, so that first call always precedes any execution-state mutation and sees exactly the locked row. Later loadState calls read from the database, so bundles of transitions still validate against the latest state. Tx.Current lets a callback inspect state without consuming the seed; the HTTP handlers, the task starters, and the worker's acquisition and abandon paths use it instead of re-reading the chat under the lock. Publication keeps reading the committed row after the commit write. Regression tests pin the single-write contract and the round-trip budget: a transition validates from the seed and updates the chats row exactly once, a metadata-only callback still advances snapshot_version once, and a bundle re-reads exactly once after the seed is consumed. Emulation (48 clients, same host, pgbench with the generated statements): LWLock:BufferContent fell from ~30% to ~15% of backend samples, insert and exec-state latencies dropped by a quarter, and organizations left the hot page list. Remaining shared-page work is the per-message model config FK check and index maintenance on the commit write.
Remove the unused ChatMachine.Lock, rename the transition lock's cached row, and rewrite comments that described what instead of why. Document that snapshot_version now advances once per commit write and that FOR SHARE still blocks the transition lock.
…on callers These six queries lost their last production callers in the chatd state machine refactor and write chat_messages or chat_queued_messages without setting the commit write's history and queue flags. Tests that seeded queue rows now use InsertChatQueuedMessageWithCreator.
2ccb0e5 to
d304912
Compare
After a lock wait, Postgres re-reads only the locked chat row, so has_queued computed in the lock statement could miss queue changes committed during the wait. Count the queue in its own statement after the lock instead.
|
@codex review |
|
To use Codex here, create a Codex account and connect to github. |
|
/coder-agents-review |
|
Chat: Review posted | View chat Review history
deep-review v0.13.0 | Round 1 | Last posted: Round 1, 9 findings (7 P3, 1 P4, 1 Nit), COMMENT. Review Finding inventoryFindings
Contested and acknowledgedNone. Law analysisRound 1. Head 7aa2856. Effective additions 1135. Verdict: Split. Enforcement: Mandatory. Round logRound 1Netero plus Law. Law returned a mandatory split, so the panel was skipped. 7 P3, 1 P4, 1 Nit, 3 out of scope. Orchestrator moved hook_server.go:85 and generation.go:427 to out of scope (unchanged code, other instances of the pattern). Reviewed against 05eac08..7aa2856. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
This PR moves every chats-row write in a transition into one commit write, drops the history and queue-version triggers, weakens the chat row locks to FOR NO KEY UPDATE, and seeds callbacks from the locked row. This round is a first pass only: mechanical findings plus a split assessment. The full review panel has not reviewed the PR yet and will review once the split lands. Findings: 7 P3, 1 P4, 1 Nit.
Split required. The PR carries four changes that can land and be reviewed separately, in this order:
- Remove the six queue and message queries with no production callers and
ChatMachine.Lock(commit d304912 and the related deletions). No behavior change. - Change the chat row locks (
GetChatByIDForUpdate,LockChatByID, the context-resource sync CTE, and the transition lock) to FOR NO KEY UPDATE, with a test showing a heartbeat upsert completes while a transition holds the lock and a second transition still blocks. On main,LockChatAndBumpSnapshotVersiontakes FOR UPDATE in its subquery, so this change works there without the rest of the PR. - Single commit write: migration 000614, the commit-write queries, the history and queue flags in
transitions.go,machine.go, andautomation.go. - Validate from the locked row:
lockedRow,Tx.Current, the queue count after the lock, and the pre-read removals inchatd.go,tasks.go,generation.go, andworker.go.
The lock change is the required split: it changes which writers exclude each other onchats, no test pins either the new non-blocking behavior or the exclusions that must remain, and inside this diff it is four keywords. Splitting 1 and 4 out is recommended. A layer split of item 3 does not work, because dropping the triggers without the flag code leaveshistory_versionandqueue_versionstale.
Out of scope (needs a ticket or explicit acceptance by a human):
coderd/x/chatd/hook_server.go:85:handleAPIDispatchErrorstill reads the chat withstore.GetChatByIDinsidemachine.Update, adding a round trip under the row lock.coderd/x/chatd/generation.go:427:syncMemoryIndexstill callsloadChatForGeneration(ctx, store, ...)insidemachine.Update, adding a read under the lock on every turn that commits a memory index message.coderd/x/chatd/chatstate/family.go:78:SetFamilyArchivedlocks each member withGetChatByIDForUpdateand again withLockChatForTransition, so every member pays two lock statements.
coderd/x/chatd/chatstate/family.go:119
P3 [CRF-2] SetFamilyArchived calls state.loadState(), which now consumes the locked row. (Netero)
Each member that needs archiving then pays an extra GetChatByID and CountChatQueuedMessages under the lock (a scratch test counted 3 and 3, against 2 and 2 with Current()). A member already at the requested value returns current, and Update's bump-only write runs after the callback, so the caller gets SnapshotVersion=3 while the database holds 4. Use state.Current() here, as the other callbacks in this PR do.
🤖
coderd/x/chatd/chatstate/transitions_test.go:41
P3 [CRF-7] The assertion message "history_version set by trigger after initial insert" is false after this PR. (Netero)
The creation commit write (BumpChatSnapshotVersion with HistoryChanged: true) sets it, and migration 000614 drops the trigger. Change the message to say so.
🤖
🤖 This review was automatically generated with Coder Agents.
| return err | ||
| } | ||
| chat, state, err := tx.loadState() | ||
| if tx.commits == 0 { |
There was a problem hiding this comment.
P3 [CRF-1] Update issues the bump-only commit write only when tx.commits == 0, so a history or queue flag set after an earlier commit write in the same callback is dropped. (Netero)
A scratch test running Acquire and then CommitStep in one Update ended at snapshot_version=2, history_version=1, with the inserted message at revision=3, so the generation history fence does not move and no error is reported. No production callback runs a flag-only transition after a commit write today. Bump when tx.commits == 0 || tx.historyChanged || tx.queueChanged, or return an error for pending flags as requireNoVersionFlags does.
🤖
|
|
||
| `chats.history_version` stores the latest `snapshot_version` in which chat message history changed. It starts at `0`, remains unchanged for non-history transitions, and is set to the current `snapshot_version` whenever a message is inserted or meaningfully updated. A newly created chat starts with `snapshot_version = 1`; because `Create` inserts initial history in that snapshot, the created chat's `history_version` becomes `1`. No-op message updates do not advance message `revision`, advance `history_version`, or reset `generation_attempt`. Whenever `history_version` changes, `generation_attempt` is reset to `0`; generation attempts are scoped to the current history version. | ||
|
|
||
| TODO: the invariant is now the reverse: the `BEFORE INSERT`/`BEFORE UPDATE` trigger stamps `revision = snapshot_version + 1`, the version the enclosing transaction commits, and every transaction that writes `chat_messages` must end with a commit write that advances `snapshot_version` and sets `history_version` (the `AFTER` triggers that wrote `history_version` and reset `generation_attempt` were dropped in migration 000608). Rewrite this paragraph and the trigger listing below accordingly. |
There was a problem hiding this comment.
P3 [CRF-3] This TODO says the triggers were dropped in migration 000608, but this PR drops them in 000614_chat_single_transition_write; 000608 is 000608_chat_title_source. (Netero)
The same number is wrong at line 394. Change both to 000614.
🤖
|
|
||
| We will not define the SQL queries that correspond to each transition - it'd take too much space and it's not central to the document's purpose. Instead, we focus on what each transition does to the database state, and how it affects the execution and ownership states. | ||
|
|
||
| TODO: the transition lock no longer writes the row. Each transition ends with a single commit write (`UpdateChatExecutionState`, `BumpChatSnapshotVersion`, `IncrementChatGenerationAttempt`, or `UpdateChatRetryState`) that advances `snapshot_version` together with the transition's own fields, and `ChatMachine.Update` issues a bump-only commit write when the callback performed none. Rewrite this paragraph and the two that follow: `history_version` and `queue_version` are now written by the commit write (from write-intent flags set by the message and queue helpers), not by triggers. The motivation is that Postgres re-runs every FK check on an UPDATE whose old row version was written by the same transaction, and those checks key-share the parent rows shared by all chats (users, organizations, chat_model_configs). |
There was a problem hiding this comment.
P3 [CRF-4] The PR adds five TODO notes next to architecture paragraphs it makes false, instead of rewriting them. (Netero)
Line 113 still says snapshot_version advances immediately after locking and before mutating any tables, and lines 117 to 119 and the trigger listing from line 396 still describe trigger-maintained history_version and queue_version. Root AGENTS.md requires updating the architecture document when chatd architecture changes, and these TODOs link no ticket. Rewrite the affected paragraphs in this PR.
🤖
| SELECT * | ||
| FROM chats | ||
| WHERE id = @id::uuid | ||
| FOR NO KEY UPDATE |
There was a problem hiding this comment.
P3 [CRF-5] No test checks that the transition lock lets FOR KEY SHARE child writes, such as heartbeat upserts, through. (Netero, Law)
rg 'NO KEY|KEY SHARE' -g '*_test.go' returns nothing, and TestConcurrentUpdatesSerializeOnChatRow only covers transitions serializing against each other, so reverting this lock to FOR UPDATE passes every test and brings back the heartbeat convoy. Add a test that holds an Update callback open and requires a heartbeat upsert for the same chat to finish before the callback is released.
🤖
| if count != 1 { | ||
| return nil, xerrors.Errorf("delete unpromotable queued message %d: deleted %d rows, want 1", row.ID, count) | ||
| } | ||
| tx.queueChanged = true |
There was a problem hiding this comment.
P3 [CRF-6] No test covers tx.queueChanged = true in dropUnpromotable. (Netero)
With this line removed, go test ./coderd/x/chatd/chatstate/ -count=1 still passes. When the guard drops every queued row, this flag is the only thing that advances queue_version, so without it stream consumers keyed on queue_version would not refetch (inferred, not run). Assert that QueueVersion advances in the existing guard-drop tests.
🤖
| err := machine.Update(ctx, func(tx *chatstate.Tx, _ database.Store) error { | ||
| // Row already locked by ChatMachine.Update (LockChatForTransition). | ||
| chat, _, err := tx.Current() | ||
| if err != nil { |
There was a problem hiding this comment.
P4 [CRF-8] The ErrChatNotFound branch after tx.Current() cannot run, so mismatch = true never fires for a deleted chat. (Netero)
On the first call Current() returns the row LockChatForTransition returned with a nil error, and a missing chat makes Update return ErrChatNotFound before the callback runs. The same unreachable branch is in loadLockedChatForTask, acquireCandidate, and abandonAcquiredChat. Delete these branches, or handle the deleted chat at the Update call site.
🤖
| // TestReadSnapshotNotBlockedByRowLock verifies that a ReadSnapshot | ||
| // completes while another transaction holds the chat row's FOR UPDATE | ||
| // lock. The former FOR SHARE read would have queued behind it. | ||
| // completes while an Update holds the chat row's transition lock. The former FOR SHARE read would have queued behind it. |
There was a problem hiding this comment.
Nit [CRF-9] This doc comment line is 121 characters, while the rest of the file wraps comments at about 72. (Netero)
Wrap it to match.
🤖
… after the commit write Update skipped its bump once a transition had committed, silently dropping later staged changes. It now returns ErrStagedAfterCommitWrite, and every matrix transition is tested in the after-commit position.
…commit writes Helpers that write messages or queue rows without a commit write are now stage* and live in staged.go; the commit write helpers live in commit.go as commitExecutionState. No behavior change.
SetFamilyArchived classifies members with Current so SetArchived validates against the locked row instead of re-reading the chat and recounting the queue. Remove not-found branches that cannot follow tx.Current, pin the FOR NO KEY UPDATE lock mode with a heartbeat insert test, assert that guard drops advance queue_version, and update the comments, test names, and database comments that still described the dropped version triggers. Point the architecture doc TODOs at migration 000614.
Discussing some potential changes to this with @hugodutka on slack, will update description to match reality once a path forward is decided.
Extended AI description
Postgres re-runs every foreign key check on an
UPDATEwhose old row version was written by the current transaction, and each check takesFOR KEY SHAREon the parent row. A chat transition used to bumpsnapshot_versionat lock time and then update thechatsrow again from the message-history trigger,UpdateChatExecutionState, the attempt increment, the retry-state write, and one queue-version trigger per queued row, so every follow-up write re-checkedowner_id,organization_id, andlast_model_config_idon parent rows shared by every chat;pg_buffercacheshowed every backend queued on theusers,organizations, andchat_model_configsheap pages.Each transition now ends with exactly one
chatsUPDATE that also advancessnapshot_version(UpdateChatExecutionState,IncrementChatGenerationAttempt,UpdateChatRetryState, or the newBumpChatSnapshotVersion, whichUpdateissues itself when the callback made no commit write).LockChatForTransitionreplacesLockChatAndBumpSnapshotVersionand writes nothing; it also locksFOR NO KEY UPDATE, so foreign-key child writes such as the batched heartbeat upserts (FOR KEY SHARE) stop convoying against transitions. Migration000614drops the history and queue-version triggers; the commit write records those changes from flags set by the message and queue helpers, and the revision trigger stampssnapshot_version + 1. The lock's returned row, plus a queue count taken right after it, seeds the transaction so callbacks validate without re-reading the chat under the lock (tx.Current()), replacing the pre-reads in the HTTP handlers, task starters, and worker.Why
FOR NO KEY UPDATE:chatsis the parent row ofchat_heartbeats,chat_messages,chat_queued_messages, and child chats, so inserting or updating those takesFOR KEY SHAREon the chat row for the foreign key check.FOR UPDATEconflicts withFOR KEY SHARE, so heartbeat upserts waited behind every in-flight transition.FOR NO KEY UPDATEdoes not conflict withFOR KEY SHAREbut still conflicts with itself,FOR UPDATE, andFOR SHARE, so transitions still serialize. It is safe because no transition changeschats.id.GetChatByIDForUpdate,LockChatByID, and the context-resource sync switch too so they keep the same mutual exclusion withUpdate.Behavior changes for callers:
snapshot_versionadvances once per commit write, not once perUpdate, so a composite with two commit writes advances it by two.Update's bump-only commit (for example a bareCommitStep) and reads the chat inside the callback sees the pre-commit versions.applySessionStartResponse,commitGenerationStep, andfinishGenerationTurn's continue path do this; perf(coderd): publish and return the chat row from the transition's commit write #30268 replaces those reads with the commit write's result.chatstatehelpers that set the commit write's flags, not the raw store.AcquirethenCommitStep) now fails withErrStagedAfterCommitWriteand rolls back. Recording those changes would need a secondchatsUPDATE, which brings back the foreign key re-checks this PR removes; before this check the changes were silently left unrecorded (history_versionandgeneration_attemptstayed put while the message carried a revision the chat never reached). No production callback does this.Why the queue is counted in its own statement: an earlier revision returned
has_queuedfrom anEXISTSsubquery inLockChatForTransitionto save a round trip. When that statement waits for the row lock, Postgres re-reads only the locked chat row; the subquery keeps the snapshot from before the wait. A queue change committed by the transition we waited behind was then invisible, so for exampleFinishTurnclassifiedR1asR0and parked the chat inwaitingwith a non-empty queue. Main'sTestQueuePromotionGuard_Concurrencycaught this. A separateCountChatQueuedMessagesafter the lock sees everything committed before the lock was granted, and its correctness does not depend on how other code writes the queue.Staged writes and commit writes: the helpers that write
chat_messagesorchat_queued_messageswithout touching thechatsrow are namedstage*and live inchatstate/staged.go(the automation promotion guard's are inautomation.go);commitExecutionStateand the flag bookkeeping live inchatstate/commit.go.TestTransitionMatrix_AfterCommitWritereruns every matrix case as if another transition had already committed, so any transition, including a future one, that stages without committing itself is rejected there (todayCommitStep,DeleteQueuedMessage, andPromoteQueuedMessagewhen the guard drops a stale automation row). A newchatdcallback that composes transitions in the wrong order is caught by any test that runs it; writes that bypass thestage*helpers are not caught.Round trips under the row lock for a single-transition
Update: main issues 7; this PR issues 5 (lock, queue count, commit write, and the publication's chat read and queue count). #30268 returns the row and queue state from the commit write, dropping the publication reads for 3 in total. A runnable result adds the heartbeat staleness check in each case.Where a transition still writes the
chatsrow more than once:SendMessageissues raw metadata UPDATEs in the same transaction when the request changes plan mode (UpdateChatPlanModeByID) or MCP servers (UpdateChatMCPServerIDs), and on a chat's first user turn (fallbackUpdateChatTitleByID, written after the commit write). A send to an unowned running chat with interrupt runs two commit writes (sendMessageQueueAndSetStatus, thenFinishInterruption). These are left as they are: they run once per user action, not per generation step or heartbeat, so they were not part of the contention this fixes. Folding them in would mean carrying plan mode, MCP server IDs, and the title through the commit write.Database impact: in a pgbench emulation of the generated statements (48 clients, same host)
LWLock:BufferContentfell from ~30% to ~15% of backend samples, insert and execution-state latencies dropped by about a quarter, andorganizationsleft the hot page list. Scaletest data had transitions holding the row lock for 20-38 ms across 7-10 round trips; this removes the pre-callback chat re-read, and #30268 removes the post-callback reads.Migration note:
000614only drops triggers and replaces a trigger function, so it is quick. Replicas still on the previous version keep working against the new schema, but their message and queue writes no longer advancehistory_version/queue_version(those triggers are gone), so keep the mixed-version window of a rolling deploy short.Main drift worth a look:
UpdateChatExecutionStatekeeps #30241'sretry_stateclearing on leavingrunning(grant_history_epochis folded intohistory_changed), and #30243'sfinishUnownedInterruptionnests a fullFinishInterruptioninsideInterrupt, so that composite is two commit writes; the transition matrix test now owns running chats in a seeder instead of inside the measuredUpdate. The automation promotion guard'sdropUnpromotabledeletes queued rows, so it now sets the queue-changed flag (the queue-version trigger used to record those deletes).Combined with the other PRs: #30268, #30269, and #30270 are based on this branch; if #30265 has merged first, the rewritten
Updatebody keeps its phase-timing boundaries aroundLockChatForTransition.Series: #30264 harness jitter, #30265 transition metrics, #30266 lock-free stream reads, #30267 single-write transitions, #30268 commit-write result and RBAC cache, #30269 non-blocking acquisition, #30270 finish turn in the step commit.
Generated by Coder Agents on behalf of @cstyan.