Repository navigation
fix(coderd/x/chatd): make compaction fit an overflowing context - #30766
Conversation
Compaction is the recovery path for a chat near its context window, but its summary request sent the whole history, the tool definitions, and a larger output cap than the turn that overflowed. The tool-less retry only ran for HTTP 400 rejections, so a context-window error reported inside an HTTP 200 stream (OpenAI Responses API, Bedrock Mantle) wedged the chat: every turn and every /compact failed the same way. - Detect context-window rejections reported inside a stream: fantasy stream errors without a status code and Responses error events. - When the request without tools is still too large, retry with the largest tool results cut to a common size. Send that request first when the full request would leave under 16k tokens for the summary. - Classify a conversation that still does not fit as a non-retryable config error that tells the user how to recover. - Count the messages after the last step, usually its tool results, in the compaction trigger, so a large tool result compacts the history before it reaches the model. - Cap a single tool result at 256 KB on large-context models. - Record the summary request shape on the compaction span and log each rejected request.
…igger Follow-ups from a review of the compaction overflow fix: - Count pending user messages in the compaction trigger, so a large pasted message compacts the history before it reaches the model. The summary request still excludes them because they follow the compacted history. - Log only summary requests the model rejects for their size, under the span attribute keys, and stop after a rejected elided request. - Rename the span attributes to summary_requests_sent and summary_request_kind, type the request kind, and unexport the fields GenerateCompaction derives. - Tell the user that editing a message deletes every message after it. - Cover an elided request on a summary window smaller than the history, a non-transient in-stream error, the rejection log, and the 256 KB cap.
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c5051f754a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…timate The compaction trigger and the summary estimate counted only the messages after the last assistant message. An interrupted step persists its partial response without usage, so the estimate skipped that response and every message between it and the step that reported the usage. Both now start after the assistant message whose step reported the usage. isContextTooLargeError now trusts the error code fantasy appends to an OpenAI Responses failure. A rate limit or server error whose text mentions the context window no longer turns into a non-retryable compaction error. Also adds tests for when the elided summary request goes first and for case-insensitive overflow wording.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d39ed3aa67
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… windows The elided summary request halved the summary model's window for the history, then added the summary prompt and the pre_compact hint. With a 16 KiB hint in a window under about 13k tokens, the smallest request could not fit. The history now gets half of the window those leave. Tool error results are now elided like text results and stay errors, so large error output no longer makes a conversation unrecoverable. Text and other non-image, non-PDF attachments now count by size in the compaction estimate. Images and PDFs keep the per-part estimate because providers bill them by dimensions and pages.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4bc6ea17c1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ummary When a compaction override model omits the chat's tool definitions, the elide-first check still counted them through the trigger step's usage. Large definitions could make compaction elide tool results before it tried the full history, which would have fit without them. The check now leaves the omitted definitions out before it elides. While the full history fits either way, the output cap still counts them, which keeps a margin for providers that reject input plus max_tokens over the window.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1dd6e54bad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…s twice When a step reports only TotalTokens and OutputTokens, the context tokens fall back to the total, which already includes the output. The summary estimate added the output again, so it could elide tool results that fit. A tool call that publishes attachments persists them as an assistant message without usage, and the prompt omits that message. The trigger counted it as an unmeasured assistant message, so the trailing estimate started before the measured step and counted its prompt again. Only assistant messages that the prompt replays now count.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 404cbd310a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…izing A summary override model sanitizes the prompt, and that can drop an assistant message, such as an interrupted OpenAI response that holds only reasoning. The summary estimate then applied the original count of unmeasured assistants to the sanitized prompt. It counted history that the measured step already covers, so the estimate could exceed the window and elide tool results that fit. The preparer now estimates the trailing input from the original prompt and passes it as SummaryTrailingTokens.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c9b39d1085
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…rompt The trigger left out the measured step's output, which the next request replays, so a large tool call could push that request past the threshold without compacting first. The trigger now adds the output to the trailing estimate, as the context gauge in the UI does. The post-compaction over-limit check still reads only the measured prompt. The prompt for an Anthropic model drops an assistant message that holds only provider tool history it cannot replay, such as an interrupted web search call without its result. latestPromptUsage still counted that message as unmeasured, so the trailing estimate counted the measured step's history again and compacted early. It now sanitizes each unmeasured message the same way before it counts it.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 83c907a89f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… the window When the summary requests omit the tool definitions, GenerateCompaction subtracted their estimate only when that alone cleared the elide-first check. Near the window the definitions then stayed in the estimate, so the output cap could not clamp to the room the tool-less request leaves: a 110k estimate with 20k of omitted definitions in a 100k window kept the configured cap instead of about 10k. Subtract them whenever the elide-first check fires and decide it again. They still stay in the estimate as a margin while the summary has room to spare.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af7d38f429
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: af7d38f429
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…allback The elided summary request now cuts large text attachments the same way it cuts local tool results, keeping their filename and media type. Images and PDFs are left as they are. A history whose size comes from attached documents can now be summarized without the user editing earlier messages.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 81980c2689
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…llback Providers that reject text attachments get them inlined as user text before compaction runs, so the elided request now also cuts large text parts in user messages. System and assistant text are not cut.
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
…imating them Drop the token estimates this branch added (the trailing-token trigger, the elide-first check, the per-request summary cap, and the elision budget) and the 256 KB tool result cap. When the model rejects a summary request for its size, the next request is thinner: first without tool definitions, then with tool results cut to 32 KiB, then to 4 KiB. A step that cuts nothing is skipped. Only locally executed text and error tool results are cut. The in-stream rejection match, the non-retryable recovery error, and the Warn log for each rejected request stay.
jscottmiller
left a comment
There was a problem hiding this comment.
Left a small note on the user message but otherwise looks good. Approach is simple and smart. I can see an argument for including large attachments in the cut as well, but those are imo less likely than big tool call outputs. I also wonder if it is worth biasing larger cuts toward the older messages, but that would need a eval to properly assess.
When even the smallest summary request is rejected for its size, the recovery message now offers /clear first. It keeps the transcript and starts the next message with a fresh context, without deleting messages or leaving the conversation.
Compaction is the recovery path for a chat near its context window, but its summary request was larger than the turn that overflowed: the whole history plus the tool definitions. Its only fallback, a retry without tools, also missed context-window errors that arrive inside an HTTP 200 stream (OpenAI Responses API, Bedrock Mantle). In the incident behind this PR, one MCP tool result of about 1 MB moved a 1M-token chat from 67% to 92% of its window in one step. After that, every turn and every
/compactfailed with "Your input exceeds the context window of this model."This PR does not try to predict whether the summary request fits. Each time the model rejects it for its size, the next request is thinner:
A step that would cut nothing is skipped. Only locally executed text and error tool results are cut. User messages, attachments, provider-executed results, and media stay whole. The cuts apply only to the summary request, not to the stored history.
Other changes:
errorandresponse.failedevents now count as size rejections. A Responses error code other thancontext_length_exceedednever matches, so a rate limit stays retryable. Anthropic's "input length andmax_tokensexceed context limit" also matches.Not in this PR: counting tool results that arrive after the measured step in the compaction trigger, estimating the summary request before sending it, and a tighter per-result cap for large-window models. Earlier revisions of this branch did these and were dropped for the simpler path above.
Testing
go test ./coderd/x/chatd/...(20 packages),golangci-lintonchatdandchatloop, and the pre-commit hook pass.httptestfor each in-stream error shape and must retry without tools.