Skip to content

chore(coderd): store a delivery mode on queued chat messages - #30073

Merged
ethanndickson merged 2 commits into
mainfrom
ethanndickson/chat-queue-delivery-mode
Nov 9, 2026
Merged

ethanndickson merged 2 commits into
mainfrom
ethanndickson/chat-queue-delivery-mode

Conversation

@ethanndickson

@ethanndickson ethanndickson commented Sep 29, 2026 •

Copy link
Copy Markdown
Member

Queued chat messages now store a delivery mode. Later in this stack, steer messages go in before the next model call instead of at turn end, so chatd needs to know which queued rows are steer rows.

The migration adds the enum chat_busy_behavior (queue, steer) and the column chat_queued_messages.busy_behavior. It's NOT NULL DEFAULT 'queue', so existing rows get queue, and chatstate still passes queue on every insert for now, so this doesn't change behaviour. The old InsertChatQueuedMessage query doesn't set the column and just gets the default, but only tests use it.

There's no interrupt value on purpose. An interrupt isn't a delivery mode, it's something that happens to the chat, and #31174 stops the API accepting it as a busy behaviour.

This supersedes #29783.

Part of CODAGT-1140

@linear-code

linear-code Bot commented Sep 29, 2026

Copy link
Copy Markdown

CODAGT-1140

@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-04T10:35:05.267568Z 2cfb041 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 1cd93a1972

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ThomasK33 added a commit that referenced this pull request Oct 2, 2026
… guard (#30096)

Adds the backend-only seam that lets a later `Publish` operation admit
an automation message inside chatd's existing create-chat and
send-message transactions, and the guard that drops stale automation
input when the queue promotes it. Only Go test callbacks use the seam in
this PR, so ordinary chat behavior does not change.

Closes CODAGT-1208
Part of CODAGT-1135

> Stack: PR 2, on top of #30085 (schema, RBAC resource, and audit
entries). No migration changes; this PR adds and extends sqlc queries
only.

ARCHITECTURE.md prose was drafted by an agent at the user's request; the
user will review it.

## Changes

- **Admission seam.** `chatstate.AdmitFunc` is a nil-default `AdmitInTx`
field on `chatd.CreateOptions`, `chatd.SendMessageOptions`,
`chatstate.CreateChatInput`, and `chatstate.SendMessageInput`.
`CreateChat` runs it after inserting the chat and before the initial
history, and requires exactly one initial user message. `SendMessage`
runs it on the locked chat before any write. The callback returns an
`AutomationProvenance` (automation ID, input ID, queue generation),
which is validated and stamped on the message. A callback error rolls
back the whole transaction, so no chat, message, or queue row survives.
`Server.CreateChat` and `Server.SendMessage` keep their signatures.
- **Provenance.** `chatstate.Message.Automation` is written to queued
rows (`automation_id`, `input_id`, `queue_generation`) and history rows
(`automation_id`, `input_id`). `InsertChatMessages` and
`InsertChatQueuedMessageWithCreator` gain the matching parameters, and
`messageFromQueuedRow` copies them on promotion.
- **Promotion guard.** `chatstate.LockAutomations` is the only way to
lock automation rows: it locks them `FOR UPDATE` in ascending ID order,
running as chatd. The chatd subject gains read on `chat_automation`. A
queued automation row passes only if its automation exists, is enabled,
and has the row's `queue_generation`. Two helpers apply the check:
- `nextPromotableQueueHead` serves `sendMessageE1`, `FinishTurn`, and
`FinishInterruption`. A head without an automation costs no extra query.
Otherwise it locks every automation in the queue up front, deletes
failing heads until one passes, and leaves rows behind that head alone.
If the queue empties, `FinishTurn` and `FinishInterruption` land in
`waiting`, as from `R0` and `I0`.
- `guardQueuedRows` serves explicit `PromoteQueuedMessage`. A failing
row is deleted and the transition returns `Rejected` without an error,
so the delete commits. `Server.PromoteQueued` then returns
`ErrQueuedMessageNotFound` (HTTP 404) without reordering, substituting a
message, or changing status.
- The steer paths (`DeliverSteerMessages` and the steer batches from
#30073/#30074) add one `guardQueuedRows` call each, whichever PR lands
second.
- **Deadlocks.** `database.IsDeadlockError` is new. `ChatMachine.Update`
reruns the transaction, at most 3 attempts with a fresh publish buffer
each, only when an attempt that took automation locks hit a deadlock
abort. Attempts without automation locks, and nested Updates, never
retry. The `SendMessage` and `PromoteQueued` closures reset their
results so a rerun is safe.
- **Transition matrix.** Adds the outputs the guard can produce, such as
`R1 -> FinishTurn -> W` and `E1 -> SendMessage -> R0`.
- **`ARCHITECTURE.md`.** Describes the admission seam, provenance, the
queue promotion guard, the lock order, and the deadlock retry, and adds
the guard's outcomes to the state diagram.

## Validation

- Go tests on real PostgreSQL cover the ticket's checks:
  - A callback error leaves no chat, message, or queue row.
- A guard rejection deletes only the rejected heads, on every promotion
path and for each failure reason (missing, disabled, disabled with the
same generation, generation mismatch).
- Explicit promotion of a failing row returns not found, and the delete
is committed.
  - Provenance round-trips through queue and history.
  - `LockAutomations` works through dbauthz.
- `Update` retries only when an attempt that took automation locks hits
a deadlock.
- A concurrency test of acceptance, promotion, and disable across mixed
automation IDs sees zero deadlock aborts.
- Mutation checks confirmed that the guard, retry, and permission tests
fail when their behavior is removed.
- Remote dogfood UAT passed on a real dev instance: sending, queueing,
head and explicit promotion, interrupting, and deleting queued messages
all behaved as before.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model:
`anthropic:claude-opus-5-5` • Thinking: `high`_
@ethanndickson
ethanndickson marked this pull request as draft October 7, 2026 12:05
@ethanndickson
ethanndickson force-pushed the ethanndickson/chat-queue-delivery-mode branch 3 times, most recently from 632582f to c140e42 Compare October 22, 2026 03:54
@ethanndickson
ethanndickson removed this pull request from stack #30075 October 22, 2026 03:56
@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@ethanndickson
ethanndickson marked this pull request as ready for review October 22, 2026 05:09
@ethanndickson
ethanndickson force-pushed the ethanndickson/chat-queue-delivery-mode branch from c140e42 to 028f65a Compare October 22, 2026 08:49
@@ -0,0 +1,33 @@
-- Take the lock this migration needs before changing anything, retrying
-- short waits so that concurrent chatd traffic cannot deadlock with it. See
-- 000609_chat_automations.up.sql for the reasoning.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove the lock-retry block. Every pending migration runs in one transaction, so if an earlier one already holds chats, chatd requests hang behind the block for 2 minutes and then the upgrade fails. Without the block, Postgres cancels one request after 1s and the upgrade succeeds. My agent reproduced this, and 000615 already fails this way on upgrades from 610.

@ethanndickson ethanndickson Oct 22, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh okay interesting. I copied it verbatim from prior migrations so i assumed it was fine?

@mafredri mafredri Oct 22, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I understand it, it's fine on dogfood due to deploying every migration individually, but can fail on release.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ethanndickson on some further analysis, this method of locking trades one issue for another. But essentially, my previous statement was wrong.

If there's a release, and first migration that locks is 509, that's the migration that can cause a problem. All future migrations that do the same lock are essentially no-op because that lock is retained throughout the migration.

See #31167 for more context.

Essentially, I think we shouldn't do this type of locking at all, and find a better way to deal with issues resulting from rolling upgrades. Personally I would like to see a method where "writes" are revoked from old replicas during upgrade.

@ethanndickson
ethanndickson force-pushed the ethanndickson/chat-queue-delivery-mode branch from 5e23b29 to a1439bd Compare October 22, 2026 10:57
@ethanndickson
ethanndickson force-pushed the ethanndickson/chat-queue-delivery-mode branch 3 times, most recently from 802a23e to aba1026 Compare October 27, 2026 00:46
@ethanndickson
ethanndickson force-pushed the ethanndickson/chat-queue-delivery-mode branch 3 times, most recently from a97485f to 2cfb041 Compare November 3, 2026 05:56
@ethanndickson

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 2cfb041a95

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ethanndickson
ethanndickson force-pushed the ethanndickson/chat-queue-delivery-mode branch from 2cfb041 to 61262fd Compare November 5, 2026 06:41
Add a chat_busy_behavior enum ('queue', 'steer') and a busy_behavior
column on chat_queued_messages. Every current insert stores 'queue', so
behavior does not change. A later change delivers 'steer' rows at the
next step boundary.
@ethanndickson
ethanndickson force-pushed the ethanndickson/chat-queue-delivery-mode branch from 61262fd to 2c514df Compare November 6, 2026 06:03
@ethanndickson
ethanndickson merged commit 8230004 into main Nov 9, 2026
90 of 97 checks passed
@ethanndickson
ethanndickson temporarily deployed to main November 9, 2026 04:55 — with GitHub Actions Inactive
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 9, 2026
@ethanndickson
ethanndickson deleted the ethanndickson/chat-queue-delivery-mode branch November 9, 2026 04:55

This branch was previously deployed

1 inactive deployment
main — 2c514df1 Deployed Nov 9, 2026 by ethanndickson via cla #252207
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants