Skip to content

test(coderd/x/chatd): fold the effort clamp fixture into the pending-user test - #30942

Merged
ibetitsmike merged 3 commits into
mainfrom
mike/chatd-fold-effort-clamp-test
Oct 27, 2026
Merged

ibetitsmike merged 3 commits into
mainfrom
mike/chatd-fold-effort-clamp-test

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

TestBuildTurnEnvironmentClampsRequestedReasoningEffortToMax built its own database fixture (OpenAI provider, model config, chat created through chatstate.CreateChat) only to call buildTurnEnvironment, which TestBuildTurnEnvironmentSplitsPendingUserRowsFromCompactionPrompt already does with the same kind of fixture. This PR deletes the clamp test and moves its checks into the pending-user test. That test's model config now carries the clamp test's options (OpenAI User sentinel, reasoning effort default low and max medium), and its unanswered user message stores high effort. The message is the last one, so InsertChatMessages copies its effort into chats.last_reasoning_effort, the value buildTurnEnvironment reads.

Why it is safe

  • Every assertion of the deleted test has a match in the remaining one: Responses provider options with effort clamped to medium, the User sentinel, the default max output tokens, compaction tool definitions equal to BuildToolDefinitions, and the compaction summary call. The exception is require.NotNil(prepared.Compaction), which was redundant because the test already dereferenced prepared.Compaction earlier.
  • I ran 12 production mutants with go test -overlay against the old test on main and the extended test. The mutants drop the requested effort at the call site or in chatRequestedEffort, skip the clamp in ResolveReasoningEffort, map the unresolved effort in ProviderOptionsForCall, drop the config provider options, or remove the default output cap. Others replace the summary call with newCall(), drop its tool choice, raise its output cap by half or drop it, or pass nil or truncated tool definitions. Both tests fail on all 12 and pass without a mutant.

Review focus

The summary call is now checked with one equality: the call template with tool choice none. Since #31042 the summary call keeps the chat call's output cap, so the equality pins that cap too, along with the template's sampling fields, which the old field checks did not.

Validation

  • go test -count=1 ./coderd/x/chatd/: passes
  • go vet ./coderd/x/chatd/...: clean
  • golangci-lint run --fix=false ./coderd/x/chatd/...: 0 issues
  • paralleltestctx -custom-funcs="testutil.Context,chatdTestContext" ./coderd/x/chatd/...: clean
  • pre-commit hook: passed

Xum, an AI agent, opened this PR for @ibetitsmike and wrote this description.

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-20T11:48:57.340176Z 093e7be Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: e34ad0fa77

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum, an AI agent, posted this comment for @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 721e0f5df7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike marked this pull request as ready for review October 19, 2026 12:15
ibetitsmike added a commit that referenced this pull request Oct 20, 2026
…#31042)

Compaction summaries asked the provider for 1.5 times the model's
configured output cap. When an admin adds a known model, the form fills
`max_output_tokens` with the provider maximum, 128,000 for
`claude-opus-5-5`, so each summary request sent `max_tokens: 192000`.
Anthropic rejected it with HTTP 400 (`max_tokens: 192000 > 128000, which
is the maximum allowed number of output tokens for claude-opus-5-5`), so
compaction failed and the chat could not shrink its context.

`compactionSummaryCall` now keeps the chat call's cap. The 50% headroom
came from #28886, which accepted the overshoot for caps set at a
provider ceiling as a tradeoff. With catalog defaults, that case is the
common one. `GenerateCompaction` still lowers the cap to the remaining
context window when that is smaller.

Closes [CODAGT-1428](https://linear.app/codercom/issue/CODAGT-1428).

The ticket also shows "Your input exceeds the context window" on a
summary request. That is a separate failure, which #30766 handles
(merged after the ticket was filed), so this PR does not change it.

**Tests**

- New `TestActiveServer_ManualCompaction/summary keeps the configured
output cap`: a 128,000 cap reaches both the chat turn and the summary
request unchanged. On the old code it fails with `expected: 128000
actual: 192000`.
- `TestBuildTurnEnvironmentClampsRequestedReasoningEffortToMax` and the
compaction hook tests now expect the unchanged default cap (32,000
instead of 48,000). #30942 rewrites the first test and keeps a `3 *
defaultChatMaxOutputTokens / 2` expectation, so whichever PR merges
second drops that factor.

**Validation**

- `go test ./coderd/x/chatd/ ./coderd/x/chatd/chatloop/ -count=1`:
passes
- `make pre-commit`: passes
- Remote dogfood UAT against `claude-opus-5-5` with a 128,000 cap: on
`main`, manual, automatic, and compaction-model-override summaries asked
for 192,000 tokens and failed with the 400 above. On this branch, all 16
summary requests (the same flows plus an unset cap and a small context
window) succeeded, and none asked for more than its configured cap. Chat
turn `max_tokens` was the same on both. UAT ran on `9230a6e9fa1`; the
head `3633adef53e` only adds a comment edit.
- No new dependencies. The polish pass and the test audit changed no
files.
- Codex review on `3633adef53e`: no findings.

> Xum, an AI agent, opened this PR for @ibetitsmike and wrote this
description.
Resolve the conflict in turn_environment_internal_test.go with this
branch's side. #31042 keeps the chat call's output cap on compaction
summaries, so the expected summary call is now the call template with
ToolChoice none, without the raised MaxOutputTokens.
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum, an AI agent, posted this comment for @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 093e7beb49

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@linear-code

linear-code Bot commented Oct 21, 2026

Copy link
Copy Markdown

CODAGT-1445

@ibetitsmike
ibetitsmike merged commit 453e21a into main Oct 27, 2026
37 checks passed
@ibetitsmike
ibetitsmike deleted the mike/chatd-fold-effort-clamp-test branch October 27, 2026 15:11
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 27, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants