Skip to content

perf(coderd): acquire chats with a non-blocking row lock - #30269

Closed
cstyan wants to merge 2 commits into
callum/chatd-04-single-writefrom
callum/chatd-06-nowait-acquire
Closed

cstyan wants to merge 2 commits into
callum/chatd-04-single-writefrom
callum/chatd-06-nowait-acquire

Conversation

@cstyan

@cstyan cstyan commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Every replica woken by the same chat:ownership hint raced for the chat row: the winner committed Acquire while the others queued on the lock, read the new owner, and rolled back. In the scaletest each turn cost about 1.2 such waits on top of its own transactions, and the winner's next transition then queued behind the losers. Acquisition now locks with FOR NO KEY UPDATE NOWAIT (LockChatForAcquisition): a held row means another transaction is already claiming or driving the chat, so the worker skips it (ErrChatLocked). Before the lock, a lock-free GetChatTransitionState read skips chats that are not runnable, are archived, or are owned with a live lease, so a replica whose winner already committed never takes the lock at all. The authoritative check still runs under the lock, so correctness is unchanged.

Database impact: losers no longer queue on the row or roll back a locked transaction, which removes both their wait and their effect on the winner's next transition. A holder that commits re-publishes chat:ownership when it leaves the chat runnable and unowned; a holder that rolls back is covered by the periodic scan.

Combined with the other PRs: based on #30267; with #30265 skipped attempts are recorded as outcome="lock_unavailable", and with #30268 the pre-check also caches the chat RBAC object so the lock authorizes without a read; whichever merges second adds those lines.

Depends on #30267.

Series: #30264 harness jitter, #30265 transition metrics, #30266 lock-free stream reads, #30267 single-write transitions, #30268 commit-write result and RBAC cache, #30269 non-blocking acquisition, #30270 finish turn in the step commit.


Generated by Coder Agents on behalf of @cstyan.

cstyan added 2 commits October 1, 2026 20:29
Postgres re-runs every foreign key check on an UPDATE whose old row version
was written by the current transaction. A transition used to bump
snapshot_version first and then update the chats row again from the message
history trigger, UpdateChatExecutionState, the attempt increment, the retry
state write, and one queue-version trigger per queued row. Each follow-up
write re-checked owner_id, organization_id, and last_model_config_id with
FOR KEY SHARE on parent rows shared by every chat, so all transitions
serialized on the users, organizations, and chat_model_configs heap pages
(pg_buffercache showed every backend queued on those three pages).

Each transition now ends with exactly one chats UPDATE that also advances
snapshot_version: UpdateChatExecutionState, IncrementChatGenerationAttempt,
UpdateChatRetryState, or the new BumpChatSnapshotVersion, which Update also
issues when a callback performed no commit write. LockChatForTransition
replaces LockChatAndBumpSnapshotVersion and writes nothing. The revision
trigger stamps snapshot_version + 1 (the version the transaction commits);
the history and queue-version triggers are dropped and the commit write
records those changes from flags set by the message and queue helpers.
InsertChat starts a chat at version 0 so CreateChat still lands on 1/1.
UpdateChatExecutionState keeps clearing retry_state whenever the chat
leaves running.

The lock also changes from FOR UPDATE to FOR NO KEY UPDATE (as do
GetChatByIDForUpdate, LockChatByID, and the context-resource sync):
transitions never change chats.id, and FOR UPDATE conflicted with the FOR
KEY SHARE locks that foreign-key child writes such as the batched
heartbeat upserts take on the chat row, so heartbeats and transitions
convoyed against each other.

The lock's RETURNING row, extended with has_queued, seeds the Tx. The first
loadState in a transaction returns it with no round trip. This is not a
cache across writes: every transition validates before it writes, so that
first call always precedes any execution-state mutation and sees exactly
the locked row. Later loadState calls read from the database, so bundles
of transitions still validate against the latest state. Tx.Current lets a
callback inspect state without consuming the seed; the HTTP handlers, the
task starters, and the worker's acquisition and abandon paths use it
instead of re-reading the chat under the lock. Publication keeps reading
the committed row after the commit write.

Regression tests pin the single-write contract and the round-trip budget:
a transition validates from the seed and updates the chats row exactly
once, a metadata-only callback still advances snapshot_version once, and a
bundle re-reads exactly once after the seed is consumed.

Emulation (48 clients, same host, pgbench with the generated statements):
LWLock:BufferContent fell from ~30% to ~15% of backend samples, insert and
exec-state latencies dropped by a quarter, and organizations left the hot
page list. Remaining shared-page work is the per-message model config FK
check and index maintenance on the commit write.
Every replica woken by the same ownership hint raced for the chat row.
The winner committed Acquire while the others queued on the lock, read
the new owner, and rolled back; each turn cost about 1.2 such waits on
top of its five transactions, and the winner's next transition then
queued behind the losers.

Acquisition now locks with FOR NO KEY UPDATE NOWAIT
(LockChatForAcquisition). A held row means another transaction is
already claiming or driving the chat, so the worker skips it. Whoever
holds the lock either commits with the chat owned, or commits with it
still runnable and unowned and re-publishes chat:ownership; a holder
that rolls back is covered by the periodic scan.

Before the lock, a lock-free GetChatTransitionState read skips chats
that are not runnable, are archived, or are owned with a live lease, so
a replica that lost the race to a winner who already committed never
takes the row lock at all. The authoritative check still runs under the
lock, so correctness is unchanged; the pre-check only avoids
acquisitions that would certainly fail.
@linear-code

linear-code Bot commented Oct 29, 2026

Copy link
Copy Markdown

GRU-113

@cstyan cstyan closed this Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant