Skip to content

fix(coderd/x/chatd): make compaction fit an overflowing context - #30766

Merged
ibetitsmike merged 14 commits into
mainfrom
mike/chatd-compaction-fit
Oct 19, 2026
Merged

ibetitsmike merged 14 commits into
mainfrom
mike/chatd-compaction-fit

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Compaction is the recovery path for a chat near its context window, but its summary request was larger than the turn that overflowed: the whole history plus the tool definitions. Its only fallback, a retry without tools, also missed context-window errors that arrive inside an HTTP 200 stream (OpenAI Responses API, Bedrock Mantle). In the incident behind this PR, one MCP tool result of about 1 MB moved a 1M-token chat from 67% to 92% of its window in one step. After that, every turn and every /compact failed with "Your input exceeds the context window of this model."

This PR does not try to predict whether the summary request fits. Each time the model rejects it for its size, the next request is thinner:

  1. The full request: tool definitions and the whole history.
  2. The same history without tool definitions (as before).
  3. Tool results longer than 32 KiB cut to 32 KiB, keeping their head and tail.
  4. Tool results longer than 4 KiB cut to 4 KiB.

A step that would cut nothing is skipped. Only locally executed text and error tool results are cut. User messages, attachments, provider-executed results, and media stay whole. The cuts apply only to the summary request, not to the stored history.

Other changes:

  • In-stream rejections match. Fantasy stream errors without a status code and Responses error and response.failed events now count as size rejections. A Responses error code other than context_length_exceeded never matches, so a rate limit stays retryable. Anthropic's "input length and max_tokens exceed context limit" also matches.
  • A conversation that still does not fit fails as a non-retryable config error that tells the user how to recover.
  • Each rejected summary request is logged at Warn with its tool count, tool result cap, and the bytes it cut.

Not in this PR: counting tool results that arrive after the measured step in the compaction trigger, estimating the summary request before sending it, and a tighter per-result cap for large-window models. Earlier revisions of this branch did these and were dropped for the simpler path above.

Testing

  • go test ./coderd/x/chatd/... (20 packages), golangci-lint on chatd and chatloop, and the pre-commit hook pass.
  • The real OpenAI Responses client runs against httptest for each in-stream error shape and must retry without tools.
  • Each of these breaks a test: removing the cut steps, sending a cut that cuts nothing, turning error results into text, cutting provider-executed results, and dropping the in-stream match.

Xum opened this PR on behalf of @ibetitsmike.

Compaction is the recovery path for a chat near its context window, but
its summary request sent the whole history, the tool definitions, and a
larger output cap than the turn that overflowed. The tool-less retry
only ran for HTTP 400 rejections, so a context-window error reported
inside an HTTP 200 stream (OpenAI Responses API, Bedrock Mantle) wedged
the chat: every turn and every /compact failed the same way.

- Detect context-window rejections reported inside a stream: fantasy
  stream errors without a status code and Responses error events.
- When the request without tools is still too large, retry with the
  largest tool results cut to a common size. Send that request first
  when the full request would leave under 16k tokens for the summary.
- Classify a conversation that still does not fit as a non-retryable
  config error that tells the user how to recover.
- Count the messages after the last step, usually its tool results, in
  the compaction trigger, so a large tool result compacts the history
  before it reaches the model.
- Cap a single tool result at 256 KB on large-context models.
- Record the summary request shape on the compaction span and log each
  rejected request.
…igger

Follow-ups from a review of the compaction overflow fix:

- Count pending user messages in the compaction trigger, so a large
  pasted message compacts the history before it reaches the model. The
  summary request still excludes them because they follow the compacted
  history.
- Log only summary requests the model rejects for their size, under the
  span attribute keys, and stop after a rejected elided request.
- Rename the span attributes to summary_requests_sent and
  summary_request_kind, type the request kind, and unexport the fields
  GenerateCompaction derives.
- Tell the user that editing a message deletes every message after it.
- Cover an elided request on a summary window smaller than the history,
  a non-transient in-stream error, the rejection log, and the 256 KB cap.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-17T15:36:06.196984Z f1c4554 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5051f754a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/generation_preparer.go Resolved
Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
…timate

The compaction trigger and the summary estimate counted only the messages
after the last assistant message. An interrupted step persists its partial
response without usage, so the estimate skipped that response and every
message between it and the step that reported the usage. Both now start
after the assistant message whose step reported the usage.

isContextTooLargeError now trusts the error code fantasy appends to an
OpenAI Responses failure. A rate limit or server error whose text mentions
the context window no longer turns into a non-retryable compaction error.

Also adds tests for when the elided summary request goes first and for
case-insensitive overflow wording.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d39ed3aa67

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
… windows

The elided summary request halved the summary model's window for the
history, then added the summary prompt and the pre_compact hint. With a
16 KiB hint in a window under about 13k tokens, the smallest request could
not fit. The history now gets half of the window those leave.

Tool error results are now elided like text results and stay errors, so
large error output no longer makes a conversation unrecoverable.

Text and other non-image, non-PDF attachments now count by size in the
compaction estimate. Images and PDFs keep the per-part estimate because
providers bill them by dimensions and pages.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4bc6ea17c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
…ummary

When a compaction override model omits the chat's tool definitions, the
elide-first check still counted them through the trigger step's usage.
Large definitions could make compaction elide tool results before it
tried the full history, which would have fit without them.

The check now leaves the omitted definitions out before it elides. While
the full history fits either way, the output cap still counts them, which
keeps a margin for providers that reject input plus max_tokens over the
window.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1dd6e54bad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
Comment thread coderd/x/chatd/generation_preparer.go Resolved
…s twice

When a step reports only TotalTokens and OutputTokens, the context tokens
fall back to the total, which already includes the output. The summary
estimate added the output again, so it could elide tool results that fit.

A tool call that publishes attachments persists them as an assistant
message without usage, and the prompt omits that message. The trigger
counted it as an unmeasured assistant message, so the trailing estimate
started before the measured step and counted its prompt again. Only
assistant messages that the prompt replays now count.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 404cbd310a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
…izing

A summary override model sanitizes the prompt, and that can drop an
assistant message, such as an interrupted OpenAI response that holds
only reasoning. The summary estimate then applied the original count of
unmeasured assistants to the sanitized prompt. It counted history that
the measured step already covers, so the estimate could exceed the
window and elide tool results that fit.

The preparer now estimates the trailing input from the original prompt
and passes it as SummaryTrailingTokens.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9b39d1085

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/generation_preparer.go Resolved
Comment thread coderd/x/chatd/generation_preparer.go Resolved
Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
…rompt

The trigger left out the measured step's output, which the next request
replays, so a large tool call could push that request past the
threshold without compacting first. The trigger now adds the output to
the trailing estimate, as the context gauge in the UI does. The
post-compaction over-limit check still reads only the measured prompt.

The prompt for an Anthropic model drops an assistant message that holds
only provider tool history it cannot replay, such as an interrupted web
search call without its result. latestPromptUsage still counted that
message as unmeasured, so the trailing estimate counted the measured
step's history again and compacted early. It now sanitizes each
unmeasured message the same way before it counts it.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 83c907a89f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
… the window

When the summary requests omit the tool definitions, GenerateCompaction
subtracted their estimate only when that alone cleared the elide-first
check. Near the window the definitions then stayed in the estimate, so
the output cap could not clamp to the room the tool-less request leaves:
a 110k estimate with 20k of omitted definitions in a 100k window kept
the configured cap instead of about 10k.

Subtract them whenever the elide-first check fires and decide it again.
They still stay in the estimate as a margin while the summary has room
to spare.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af7d38f429

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: af7d38f429

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
Comment thread coderd/x/chatd/chatloop/tooltruncate.go Resolved
…allback

The elided summary request now cuts large text attachments the same way
it cuts local tool results, keeping their filename and media type.
Images and PDFs are left as they are. A history whose size comes from
attached documents can now be summarized without the user editing
earlier messages.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 81980c2689

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
…llback

Providers that reject text attachments get them inlined as user text
before compaction runs, so the elided request now also cuts large text
parts in user messages. System and assistant text are not cut.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: 0dd18a33aa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

…imating them

Drop the token estimates this branch added (the trailing-token trigger,
the elide-first check, the per-request summary cap, and the elision
budget) and the 256 KB tool result cap. When the model rejects a summary
request for its size, the next request is thinner: first without tool
definitions, then with tool results cut to 32 KiB, then to 4 KiB. A step
that cuts nothing is skipped. Only locally executed text and error tool
results are cut.

The in-stream rejection match, the non-retryable recovery error, and the
Warn log for each rejected request stay.

@jscottmiller jscottmiller left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a small note on the user message but otherwise looks good. Approach is simple and smart. I can see an argument for including large attachments in the cut as well, but those are imo less likely than big tool call outputs. I also wonder if it is worth biasing larger cuts toward the older messages, but that would need a eval to properly assess.

Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
Comment thread coderd/x/chatd/chatloop/compaction.go Resolved
When even the smallest summary request is rejected for its size, the
recovery message now offers /clear first. It keeps the transcript and
starts the next message with a fresh context, without deleting
messages or leaving the conversation.
@ibetitsmike
ibetitsmike merged commit 8245457 into main Oct 19, 2026
49 of 51 checks passed
@ibetitsmike
ibetitsmike deleted the mike/chatd-compaction-fit branch October 19, 2026 22:30
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants