Skip to content

fix(coderd): stop announcing stale chat retries and fence retried attempts - #30241

Merged
ThomasK33 merged 4 commits into
mainfrom
fix-chatd-stream-retry-state
Oct 1, 2026
Merged

ThomasK33 merged 4 commits into
mainfrom
fix-chatd-stream-retry-state

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Agents chats could show a "Retrying request / Attempt N" banner for a chat that was no longer running. A chat that failed showed the retry banner instead of the red "Request failed" alert, and a chat interrupted during the retry wait showed it while idle. Reconnecting did not clear it. Separately, after a retry was announced, late preview parts of the failed attempt were still streamed, which cleared the retry banner and showed a fragment that started mid-message.

Root cause

  • chats.retry_state was only cleared when generation_attempt changed (trigger sync_chat_retry_state). It survived FinishError, EnterRequiresAction, and an interrupt during the retry backoff when the failed attempt produced no parts. The stream loop then emitted retry after status/error regardless of status.
  • A retry does not change the (history_version, generation_attempt) episode, so the stream loop kept accepting parts of the failed attempt that were still in flight after the retry event. The frontend clears its preview on retry, so the next stale part replaced the banner with a partial message.

Fix

  • UpdateChatExecutionState, the only production status writer, now clears retry_state whenever it writes a status other than running. The existing trigger bumps retry_state_version, so connected streams observe the clear. No migration.
  • The stream loop only emits retry while the chat is running, as defense in depth for rows written before this change.
  • Once the stream loop observes a retry payload, or an advanced retry_state_version within an unchanged episode (a retry that was recorded and cleared again before the sync), it marks the current attempt as retired and drops its remaining parts (before the sequence checks, without an error) until the history version or generation attempt changes. The flag survives a later cancellation of the retry, since the failed attempt does not come back. The frontend is unchanged.

TestStreamLoopQueueStatusRetryErrorActionRequiredAndPreviewReset asserted error followed by retry for a chat in error, which encoded the bug. It now expects no retry there and covers retry on a running chat separately. ARCHITECTURE.md describes the new retry-state clearing, the running-status guard, and the retired-attempt rule.

How it was found

A TLA+ model of the chatd stream loop produced the counterexamples, and Go repro tests confirmed them on main. The model itself is not part of this PR.

  • TestStreamSnapshotRetryAfterTurnEnded (interrupt during backoff, error, and requires_action exits): failed before with events [message status error retry preview_reset] and [message status retry preview_reset], passes after. It also asserts retry_state is cleared and retry_state_version advanced.
  • TestStreamLoopLatePartAfterRetry: failed before with "Should be false" (stale part accepted), passes after. Subtests cover the next attempt streaming normally and a cancelled retry keeping the attempt retired.
  • TestStreamLoopLatePartAfterClearedRetry: a retry recorded and cleared before the stream syncs still retires the attempt (found in Codex review).
  • go test ./coderd/x/chatd/... -count=1 passes.

Out of scope / follow-ups

  • Rows that already hold a stale retry_state from before this change can still be announced to a newly connected client between Acquire and the next RecordGenerationAttempt, which clears it. Not worth a data migration.
  • The retired-attempt flag is set by any non-null retry_state, including a payload that fails to parse and therefore emits no retry event. The failed attempt is dropped either way; this is intentional.

Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

…empts

Two chatd stream bugs around retry events.

Retry announced for a chat that stopped running. chats.retry_state was
only cleared when generation_attempt changed, so it survived FinishError
and an interrupt during the retry backoff when the failed attempt
produced no parts. The stream then emitted retry after status/error, and
the frontend showed a "Retrying request" banner instead of the failure,
also on reconnect.

retry_state is pending-retry intent of a running turn, so it is now
cleared at the source: UpdateChatExecutionState, the only production
status writer, nulls it whenever it writes a status other than running.
The existing trigger bumps retry_state_version, so no migration is
needed. As defense in depth for rows written before this change, the
stream loop only emits retry while the chat is running.

Late preview part of a failed attempt accepted after retry. A retry does
not change the (history_version, generation_attempt) episode, so parts
of the failed attempt still in flight were accepted after the retry
event, which already cleared the client preview. The stream loop now
latches the attempt as retired once it observes a retry payload and
drops its parts before the sequence checks, until the history version
or generation attempt changes. Failed-attempt retirement is kept even
if the retry is cancelled later, since the attempt does not come back.
The server fence is sufficient, so the frontend is unchanged.

TestStreamLoopQueueStatusRetryErrorActionRequiredAndPreviewReset
asserted error followed by retry on an error-status chat, which encoded
the bug. It now expects no retry there and covers retry on a running
chat separately. ARCHITECTURE.md gets TODOs for the owner.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
…ion bump

Follow-up to the retry-state fix. TestStreamSnapshotRetryAfterTurnEnded
gains a requires_action case: a committed step leaves a pending dynamic
tool call, a retry is recorded, and the chat then enters
requires_action. retry_state must be cleared and the stream must not
emit retry. Every DB-backed case now also asserts that
retry_state_version advanced past its value after RecordRetryState,
pinning the invariant that every client-visible retry-state change
advances retry_state_version so connected streams observe the clear.

Known and intentional, left as-is:
- attemptRetired latches on any non-null retry_state, including one
  whose payload fails to parse and emits no retry event. The failed
  attempt is retired either way.
- Rows that already hold a stale retry_state from before the fix can
  still be announced to a newly connected client between Acquire and
  the next RecordGenerationAttempt, which clears it.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-01T11:52:09.122131Z f1e429b Manual request
🔒 Security Review ✅ Completed 2026-10-01T11:52:45.429504Z f1e429b Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 25d4079515

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/ARCHITECTURE.md Resolved
Comment thread coderd/x/chatd/ARCHITECTURE.md Resolved
Comment thread coderd/x/chatd/ARCHITECTURE.md Resolved
@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 25d4079515

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…pts in ARCHITECTURE.md

Replace the three TODOs added with the retry-state fix:

- Retry state version: retry_state is also cleared whenever
  UpdateChatExecutionState writes a status other than running.
- Stream loop: local state gains attempt_retired, the apply order gains
  the step that sets and clears it, and Part rejects parts of a retired
  attempt before the sequence check.
- Retry-state synchronization: retry is only emitted while the chat is
  running.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: e68d09e187

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e68d09e187

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/stream_loop.go Resolved
If RecordRetryState and a transition that leaves running (Interrupt,
FinishError, EnterRequiresAction) both commit before the stream loop
syncs, the loop only sees an advanced retry_state_version with a null
payload for the same episode, and kept accepting parts of the failed
attempt. Within one episode retry_state only changes when a retry is
recorded for the current attempt or that pending retry is cleared, so
an advanced version in an unchanged episode now also retires the
attempt. Found by Codex review; covered by
TestStreamLoopLatePartAfterClearedRetry.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: f1e429bb55

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: f1e429bb55

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33
ThomasK33 merged commit 9ac4628 into main Oct 1, 2026
33 checks passed
@ThomasK33
ThomasK33 deleted the fix-chatd-stream-retry-state branch October 1, 2026 12:10
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants