Repository navigation
fix(coderd): stop announcing stale chat retries and fence retried attempts - #30241
Conversation
…empts Two chatd stream bugs around retry events. Retry announced for a chat that stopped running. chats.retry_state was only cleared when generation_attempt changed, so it survived FinishError and an interrupt during the retry backoff when the failed attempt produced no parts. The stream then emitted retry after status/error, and the frontend showed a "Retrying request" banner instead of the failure, also on reconnect. retry_state is pending-retry intent of a running turn, so it is now cleared at the source: UpdateChatExecutionState, the only production status writer, nulls it whenever it writes a status other than running. The existing trigger bumps retry_state_version, so no migration is needed. As defense in depth for rows written before this change, the stream loop only emits retry while the chat is running. Late preview part of a failed attempt accepted after retry. A retry does not change the (history_version, generation_attempt) episode, so parts of the failed attempt still in flight were accepted after the retry event, which already cleared the client preview. The stream loop now latches the attempt as retired once it observes a retry payload and drops its parts before the sequence checks, until the history version or generation attempt changes. Failed-attempt retirement is kept even if the retry is cancelled later, since the attempt does not come back. The server fence is sufficient, so the frontend is unchanged. TestStreamLoopQueueStatusRetryErrorActionRequiredAndPreviewReset asserted error followed by retry on an error-status chat, which encoded the bug. It now expects no retry there and covers retry on a running chat separately. ARCHITECTURE.md gets TODOs for the owner. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
…ion bump Follow-up to the retry-state fix. TestStreamSnapshotRetryAfterTurnEnded gains a requires_action case: a committed step leaves a pending dynamic tool call, a retry is recorded, and the chat then enters requires_action. retry_state must be cleared and the stream must not emit retry. Every DB-backed case now also asserts that retry_state_version advanced past its value after RecordRetryState, pinning the invariant that every client-visible retry-state change advances retry_state_version so connected streams observe the clear. Known and intentional, left as-is: - attemptRetired latches on any non-null retry_state, including one whose payload fails to parse and emits no retry event. The failed attempt is retired either way. - Rows that already hold a stale retry_state from before the fix can still be announced to a newly connected client between Acquire and the next RecordGenerationAttempt, which clears it. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
|
@codex review |
|
@codex security review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 25d4079515
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…pts in ARCHITECTURE.md Replace the three TODOs added with the retry-state fix: - Retry state version: retry_state is also cleared whenever UpdateChatExecutionState writes a status other than running. - Stream loop: local state gains attempt_retired, the apply order gains the step that sets and clears it, and Part rejects parts of a retired attempt before the sequence check. - Retry-state synchronization: retry is only emitted while the chat is running. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
|
@codex review |
|
@codex security review |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e68d09e187
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
If RecordRetryState and a transition that leaves running (Interrupt, FinishError, EnterRequiresAction) both commit before the stream loop syncs, the loop only sees an advanced retry_state_version with a null payload for the same episode, and kept accepting parts of the failed attempt. Within one episode retry_state only changes when a retry is recorded for the current attempt or that pending retry is cleared, so an advanced version in an unchanged episode now also retires the attempt. Found by Codex review; covered by TestStreamLoopLatePartAfterClearedRetry. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Agents chats could show a "Retrying request / Attempt N" banner for a chat that was no longer running. A chat that failed showed the retry banner instead of the red "Request failed" alert, and a chat interrupted during the retry wait showed it while idle. Reconnecting did not clear it. Separately, after a retry was announced, late preview parts of the failed attempt were still streamed, which cleared the retry banner and showed a fragment that started mid-message.
Root cause
chats.retry_statewas only cleared whengeneration_attemptchanged (triggersync_chat_retry_state). It survivedFinishError,EnterRequiresAction, and an interrupt during the retry backoff when the failed attempt produced no parts. The stream loop then emittedretryafterstatus/errorregardless of status.(history_version, generation_attempt)episode, so the stream loop kept accepting parts of the failed attempt that were still in flight after theretryevent. The frontend clears its preview onretry, so the next stale part replaced the banner with a partial message.Fix
UpdateChatExecutionState, the only production status writer, now clearsretry_statewhenever it writes a status other thanrunning. The existing trigger bumpsretry_state_version, so connected streams observe the clear. No migration.retrywhile the chat isrunning, as defense in depth for rows written before this change.retry_state_versionwithin an unchanged episode (a retry that was recorded and cleared again before the sync), it marks the current attempt as retired and drops its remaining parts (before the sequence checks, without an error) until the history version or generation attempt changes. The flag survives a later cancellation of the retry, since the failed attempt does not come back. The frontend is unchanged.TestStreamLoopQueueStatusRetryErrorActionRequiredAndPreviewResetassertederrorfollowed byretryfor a chat inerror, which encoded the bug. It now expects noretrythere and coversretryon a running chat separately.ARCHITECTURE.mddescribes the new retry-state clearing, the running-status guard, and the retired-attempt rule.How it was found
A TLA+ model of the chatd stream loop produced the counterexamples, and Go repro tests confirmed them on main. The model itself is not part of this PR.
TestStreamSnapshotRetryAfterTurnEnded(interrupt during backoff, error, and requires_action exits): failed before with events[message status error retry preview_reset]and[message status retry preview_reset], passes after. It also assertsretry_stateis cleared andretry_state_versionadvanced.TestStreamLoopLatePartAfterRetry: failed before with "Should be false" (stale part accepted), passes after. Subtests cover the next attempt streaming normally and a cancelled retry keeping the attempt retired.TestStreamLoopLatePartAfterClearedRetry: a retry recorded and cleared before the stream syncs still retires the attempt (found in Codex review).go test ./coderd/x/chatd/... -count=1passes.Out of scope / follow-ups
retry_statefrom before this change can still be announced to a newly connected client betweenAcquireand the nextRecordGenerationAttempt, which clears it. Not worth a data migration.retry_state, including a payload that fails to parse and therefore emits noretryevent. The failed attempt is dropped either way; this is intentional.Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high