Skip to content

feat(coderd): clean up the chat queue of a deleted automation - #32140

Draft
ThomasK33 wants to merge 17 commits into
tk/automations-tool-readfrom
tk/automations-queue-cleanup
Draft

ThomasK33 wants to merge 17 commits into
tk/automations-tool-readfrom
tk/automations-queue-cleanup

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

On hold, do not review yet. No code queues a message with an automation_id until B2c (the PR that publishes automation messages), so this cleanup has nothing to clean before then. It leaves the review stack and lands together with B2c.

Automations (CODAGT-1135) must leave no work behind when they go away. The RFC says: "Disabling or deleting an automation stops new runs and deletes its queued messages in the same transaction." On the base branch the delete route refuses with an internal error while a queued message names the automation, and the soft delete of a user removes that user's automations in a trigger that does not look at the queue. With this PR both deletes remove the queued messages.

Stacked on #32079. Review only this diff. It merges after #32079.

What changes:

  • chatstate gets the transition DeleteQueuedMessagesByAutomation. It removes every queued message of the chat that one of the given automations queued. Every valid state allows it, archived chats included, and a chat without such a message does not change.
  • chatautomation.DeleteQueuedMessages runs that transition for each chat that holds such a message, in chat ID order, in the caller's transaction and as the chatd subject. The removal follows from the change to the automation, so it does not depend on what the actor can do to a chat. The function returns the publish step, which the caller runs after the commit. A disable can call it too.
  • chatautomation.Delete runs one transaction: the automation lock, the delete permission check, the queue cleanup, then the delete of the row. It reads the row count of the delete, so a row that is already gone answers 404 and writes no audit entry.
  • deleteUser takes two steps. One transaction takes the lock of each automation of the user in ID order (chatautomation.LockByOwner) and soft deletes the user, whose trigger deletes the automation rows. It locks no chat row. After the commit, chatautomation.DeleteQueuedMessagesByChat removes the queued messages by the IDs read first, one chat per transaction. A queued message keeps the ID of its automation, because chat_queued_messages.automation_id has no foreign key. A user without automations takes no lock and runs no cleanup. The cleanup runs last, after the administrators are notified and the response is written, so a client that disconnects during the cleanup changes neither the notification nor the audit status.
  • InsertChatQueuedMessageWithCreator takes automation_id and input_id. No product code sets them yet. The tests seed queued automation messages through them, and the PR that adds Publish is their first writer.
  • The chat watch event converter moves from chatd to db2sdk.ChatWatchEventChat, so that chatautomation publishes the chat list event without an import of chatd.

Lock order. Every transaction of an automation path takes the automation lock first, then chat rows in chat ID order, and writes the automation row last. The row write comes last because dbpurge deletes a chat first, and the same statement then sets target_chat_id and created_by_chat_id to NULL in the automations that name the chat. dbpurge also locks the chats of a batch in plan order, so it can form a lock cycle with a transaction that holds one chat and waits for another. This PR does not change dbpurge. A cleanup that will lock two or more chats in one transaction first takes the advisory lock of the purge job (LockIDDBPurge), which a purge tick holds from its start to its commit. A tick that finds the lock taken skips, as it does when another replica runs the purge. A cleanup of one chat takes no purge lock. The archive and unarchive of a chat family on main lock the root chat first and then its children. They keep that order: this PR changes no lock order of a main path.

deleteUser cannot follow that order in one transaction. The soft delete trigger deletes the automation rows, so it has to come after the chat locks. It also deletes the api_keys rows of the user, which dbpurge locks before it locks chats. So the soft delete commits first and holds no chat row, and each chat is then cleaned in its own transaction.

The RFC sentence "in the same transaction" holds for the automation delete. It does not hold for the delete of the owner, which the RFC does not describe. A run takes the automation lock and reads the automation again, so no run queues a message after the soft delete commits. Queued messages of the user's automations stay in three cases:

  • The cleanup of a chat fails, for example for a chat in an invalid state. The server logs a warning with the chat and the cause, cleans the other chats, and the user delete answers as on the base.
  • The server stops between the soft delete and the cleanup.
  • A user is soft deleted by direct SQL. Only the trigger runs then, with no lock and no cleanup.

In each case the chats belong to the deleted user, so nothing runs those messages: a turn in such a chat stops when it cannot create an API key for the user.

One race stays open for the create path, which is not in this base: a create that commits between the read of the automation IDs and the soft delete. The per-owner create lock of #32096 closes it. Of the two PRs, the one that merges second adds that lock to chatautomation.LockByOwner, the one place that needs it.

What a reviewer must know:

  • An invalid chat refuses the removal, as it refuses every transition but the reconcile. The automation delete then answers 409 and deletes nothing: "A chat that holds queued messages of this automation is in an invalid state. Reconcile the chat, then delete the automation again." The detail names the chat, for POST /api/v2/chats/{chat}/reconcile-invalid.
  • When the removed message is the blocked head of a paused chat, the chat leaves paused and the next queued message is promoted, as when the owner of the chat deletes that head. The caller of the cleanup need not be the chat owner: an organization admin can delete another member's automation, and a user admin can delete the user. On main the queued message endpoints are owner only, because a promotion starts a turn with the owner's credentials. Here the promoted message is one that the owner queued. This promotion writes no queue_wait trace record.
  • A cleanup does not fail because the next message cannot run. When that message has no usable model config, nothing is promoted, and the chat goes to error with its queue and an error that tells the owner to choose a model.
  • The automation delete checks the delete permission with the RBAC authorizer before the queue cleanup, and the delete of the row checks it again in dbauthz. An actor who can read the automation and cannot delete it gets 403, and no chat changes.
  • The automation delete reads the row with its webhook secret. Once that secret is encrypted with dbcrypt, the delete fails loudly on a store that cannot decrypt the row, as DeleteExternalAuthLink does on main. The user delete reads only automation IDs.
  • chatstate/transition.go is mostly gofmt realignment of the transition table. Review it with whitespace changes hidden.
  • ARCHITECTURE.md prose was drafted by an agent at the user's request; the user will review it.

Live check: none. No path queues an automation message yet, so a deployment cannot hold one. The tests seed them in PostgreSQL.

Part of CODAGT-1135


Generated with xum • Model: anthropic:claude-fable-5-2 • Thinking: xhigh

Deleting an automation, or the user who owns it, now removes the
queued messages of that automation in the same transaction. The new
chatstate transition DeleteQueuedMessagesByAutomation does the removal
under the chat row lock, so the queue version, the paused state and the
notifications follow the state machine.

The delete of an automation no longer refuses while a queued message
names it, and it reports a row that is already gone as not found. The
delete of a user takes the lock of each automation of the user, soft
deletes the user, and then cleans up the chats, in one transaction.

A chat in an invalid state refuses the transition. Both deletes then
answer 409, name the chat, and change nothing.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-fable-5-2` • Thinking: `xhigh`_
---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-fable-5-2` • Thinking: `xhigh`_
Tests for a chat that goes away during the cleanup, the order of the
chat locks and of the automation locks, a message of the same
automation in another chat, the chat update of a user delete, and a
caller who cannot delete the user and must not wait for an automation
lock.

DeleteQueuedMessages no longer returns early for an empty list. The
query finds no chat then, so the branch changed nothing.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-fable-5-2` • Thinking: `xhigh`_
Nothing reads how many queued messages the transition removed, so the
query and the transition result no longer carry the count.

The user delete test now runs as a user admin, who can read neither
the automations nor the chats of the user, and the dbauthz test checks
the two reads of the cleanup with real roles.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-fable-5-2` • Thinking: `xhigh`_
---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-fable-5-2` • Thinking: `xhigh`_
…ed message

The exhaustruct lint wants every field of the insert parameters.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-fable-5-2` • Thinking: `xhigh`_
…y chats

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-fable-5-2` • Thinking: `xhigh`_
@linear-code

linear-code Bot commented Nov 12, 2026

Copy link
Copy Markdown

CODAGT-1135

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-13T05:30:05.635177Z 6717b60 Manual request
🔒 Security Review ✅ Completed 2026-11-13T05:29:19.393161Z 6717b60 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Bravo.

Reviewed commit: e29b325741

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: e29b325741

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…annot promote

From P, DeleteQueuedMessagesByAutomation promotes the message behind a
removed blocked head. When that message has no usable model config the
promotion failed, and so did the delete of the automation or of its
owner. The transition now removes the rows and sets the chat to error
with its queue (E1). Paused is valid only with a blocked head, so the
chat cannot stay paused with a ready one.
…ion row

The purge of old chats deletes a chat row and then sets target_chat_id
and created_by_chat_id to NULL in the automations that name the chat.
Delete removed the automation row first and locked the chats second,
which formed a lock cycle with the purge. Delete now cleans the chats
first and deletes the row last.

Tests: the order against a chat delete on three connections, the publish
after the commit, and a paused chat whose next message has no model.
The transition changes the active history when it promotes from P, and
it reaches E1 when the next message has no usable model config. The
lock order gains its third step, the write of the automation row, with
the reason: the purge of old chats locks a chat row before it updates
the automations that name the chat.
DeleteOldChats deleted its batch in plan order. A transaction that
holds one chat and waits for another one, as the delete of a chat
automation does, could then form a lock cycle with the purge. The
query now locks the batch in chat ID order before it deletes it.
…mmits

The soft delete trigger removes the automation rows and the API keys of
the user. The purge of old chats locks API keys before chat rows, and
chat rows before automation rows, so no single transaction can hold all
three in an order that fits it. The delete of a user now commits the
soft delete with the automation locks and no chat row, and then removes
the queued messages one chat per transaction. A chat whose cleanup
fails is logged and keeps its messages, and the delete still succeeds.
…rge order test

A queued message insert writes a new version of the chat row, so the
table held the lower chat first and a purge in plan order met it first
too. The test now archives the chats after it queues the messages, and
it asserts that the table holds the higher chat first.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 29c66dfc90

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 29c66dfc90

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

This reverts the change that made the purge lock its batch in chat ID
order. The archive and unarchive of a chat family lock the root chat
first and then its children, so the new order formed a lock cycle with
them. The purge keeps the order that it has on main.
The purge of old chats locks the chats of a batch in the order of its
plan. A cleanup that locks two or more chats in one transaction now
takes the advisory lock of the purge job first, so the two never run
side by side. A cleanup of one chat takes no purge lock.

The delete of an automation checks the delete permission before the
cleanup, so an actor who can only read the automation gets 403 and no
conflict that names a chat.

The delete of a user runs its cleanup last, after the notifications
and the response status, so a client that goes away during the cleanup
changes neither.
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 6717b60267

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6717b60267

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/users.go
// response is written: the cleanup can wait for a chat lock, and a
// client that goes away meanwhile must change neither. A failure of the
// cleanup is logged and does not fail the request.
defer chatautomation.DeleteQueuedMessagesByChat(ctx, api.Logger, api.Database, api.Pubsub, automationIDs)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Run queue cleanup outside the request handler

When any affected chat row is locked or the user has many affected chats, this deferred call runs synchronously before deleteUser returns. Because httpapi.Write does not flush the small response explicitly, the client can remain blocked until cleanup finishes or reaches its 30-second timeout even though the user deletion has already committed; the earlier commitAudit defer is also delayed by LIFO ordering. Dispatch this best-effort cleanup to background work rather than holding the HTTP request open.

Useful? React with 👍 / 👎.

@ThomasK33
ThomasK33 changed the base branch from tk/automations-api-read to tk/automations-tool-read November 13, 2026 08:43
@ThomasK33
ThomasK33 marked this pull request as draft November 13, 2026 12:26

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant