Skip to content

feat: graduate the chat advisor from experiments - #31414

Merged
ibetitsmike merged 8 commits into
mainfrom
mike/graduate-advisor
Oct 29, 2026
Merged

ibetitsmike merged 8 commits into
mainfrom
mike/graduate-advisor

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Graduates the chat advisor: root chats get the advisor tool without the chat-advisor experiment. Plan-mode turns, child chats and explore subagents still do not get it, and the configured limits and advisor model override still apply.

  • Moves the advisor config routes from /api/experimental/chats/config/advisor to /api/v2/chats/config/advisor with Swagger docs, and promotes the SDK methods to Client. enabled is always true and read-only. The experimental /chats/config segment stays reserved, so the old paths return 404 rather than reaching the {chat} route and failing UUID parsing with a 400.
  • Removes chat-advisor from the experiment registry. The Coder Agents settings page always shows the advisor controls, and the docs drop the opt-in wording and label the Advisor page beta instead of early access, which the feature stages page defines as disabled by default.
  • Fixes max_uses_per_run, which reset on every model step because chatd rebuilds the advisor runtime per step. The runtime now counts successful advisor results since the latest user prompt, including across a compaction inside the turn. Provider errors do not use up the limit.
  • Fixes the nested advisor context. The advisor's own pending call reached it paired with a synthetic "tool call was interrupted" result, and the advisor sometimes reported that as a failure instead of answering. That result is now a pending note: the advisor is answering this call, and earlier instructions in the conversation are for the parent agent. Dropping the call instead made the advisor act as the parent on prompts such as "call the advisor tool, then report exactly what it returned". The interrupted-result defect predates the PR; remote UAT found it on main as well.
  • Corrects the advisor copy: the deployment Advisor card said 0 means unlimited for both limits, but 0 max tokens uses the server default; the limit-reached card said the limit applies to the conversation, but it is per turn.
  • Updates coderd/x/chatd/ARCHITECTURE.md line 448, which still listed the advisor routes as experimental-only.

Size: +769 and -223 lines. Generated API docs account for 280 net lines: as public /api/v2 endpoints, the two advisor routes need Swagger annotations, which experimental routes do not have. Apart from those, the graduation commit removes 81 hand-written lines. The rest is the two fixes: the use limit fix adds 50 lines of code and 171 of tests, and the pending call fix adds 51 lines of code and 77 of tests.

Delivery record

  • History: rebased onto main 3184d73 on 2026-10-29; the previous head was 251d781, on main dbd7db6. Conflicts came from feat!: choose the computer use model per organization #31300, which moved the computer-use model to organizations and removed the experimental computer-use provider routes and UI, and from fix(coderd/x/chatd): add ACP subagent prompt guidance #31498, which added ACP subagent guidance to TestBuildTurnEnvironmentToolPolicy. The advisor routes were then the last routes in the experimental /chats/config group, so the graduation commit now reserves that segment, matching the reserved segments in the /api/v2 chats routes; without it, TestChatRouteMounts failed with a 400 for GET /api/experimental/chats/config/system-prompt. Main's TestChatModelOverridesComputerUseRequiresVirtualDesktopExperiment used the removed ExperimentChatAdvisor as an unrelated experiment and now uses ExperimentAgentLifecycleHooks. Review cleanups squashed into their commits: rewrapped comments, a corrected test comment, a shorter enabled field doc, and a shared MockAdvisorConfig story fixture. Commits named below by their earlier SHAs are now: 15b689c is 8a4462a (graduation), 9dabee9 is db19b90 (per-turn limit), 72d1024 is 641ad8d (route assertion), 53dbb5c is bfba521 (first context fix), 27053d9 is d1b3d3c (test audit fixes), 3f5d0a8 is 348c390 (pending note), 799508d is c095b13 (Codex fixes) and 251d781 is e898d5b (final advisory follow-ups). A second rebase moved the branch from main 3184d73 to main 4e8c5bf, after feat: graduate MCP tool search from experiments #31327 graduated MCP tool search. Its only conflict was the generated experiment list in docs/reference/api/schemas.md, which now drops both flags; git range-diff shows every other patch unchanged.
  • Hooks: the full pre-commit hook passed on every commit. The unchanged pre-push hook ran in full (CODER_HOOK_RUN_ALL=1, SIGHUP reset to default) on each push and passed; on 3f5d0a8: make test (325s) and make test-js (82s); on 799508d: make test (102s), make test-js (81s) and the site build (29s); on 251d781: make test (198s), make test-js (94s) and the site build (35s). After this rebase, the full pre-commit hook passed on each fixup commit; under Mike's standing rule the pre-push hook was not run for the 6007ee5 and e898d5b pushes, and CI covers them.
  • Tests: go test ./coderd/x/chatd/... and the focused coderd, telemetry and codersdk advisor, route and experiment tests pass. TestAdvisorToolMarksItsOwnPendingCall pins the exact nested prompt, and TestAdvisorHappyPath_RootChat checks the real prompt pipeline for the pending note and for no interrupted result. Both fail against 27053d9 (call dropped), and the end-to-end check also fails against 72d1024 (interrupted result). On the rebased branch: go build ./..., compiling the coderd, codersdk and enterprise tests, go test ./coderd/x/chatd/..., the focused coderd route, advisor config and computer-use tests, and the telemetry and codersdk tests pass; on the site, tsc, Biome, the API and OrganizationAgentSettings Vitest suites (392 tests) and the changed stories plus the advisor tool stories (326 tests) pass. After the last change, which only moved the story fixture, tsc, Biome and the two affected story files (37 tests) pass. After the second rebase, make gen leaves the tree clean, and go build ./..., the coderd, codersdk and enterprise test builds, go test ./coderd/x/chatd/..., the focused coderd, telemetry and codersdk tests, tsc, Biome, oxlint, 392 Vitest tests and 84 story tests pass.
  • Offline replay: the recorded round-3 nested requests for the repro prompt, replayed against gpt-5-mini with only the advisor's own call changed: 20 of 20 bad answers with the call dropped, 0 of 30 at low reasoning effort and 0 of 20 at high effort with the pending note. Neutral questions were unchanged.
  • Remote UAT: round 1 found the per-turn limit defect, fixed here. Round 2 passed every graduation criterion but found the interrupted-result defect on both the branch and main (chat). Round 3 on 53dbb5c failed F1: with the call dropped, only 13 of 20 repro chats got the tip and 15 of 20 got a made-up report of the call, against 10 of 10 clean tips on the commit before (chat); everything else passed. Round 4 on 3f5d0a8 passed F1 to F6 and every edge case: the repro prompt got the tip in 16 of 16 chats at low effort and 3 of 4 at high effort (one empty reply from the known budget issue), with exactly one pending note before the question in all 46 nested requests (chat). After it, 799508d changes only a story assertion and docs line breaks, and 251d781 changes only UI copy, two comments, the docs stage label and the architecture document.
  • Reviews: 10 completed before this rebase: consolidated frontend review (no findings), UAT endorsements for rounds 1 to 4, the test audit (no blocking findings; its two should-fix gaps are fixed in 27053d9), two Codex passes, a fresh final advisory on 799508d, and a third Codex pass on 251d781. Mike raised the cap to apply the advisory's follow-ups and the architecture fix. The frontend change in 251d781 is two copy strings, so it had the FE1 to FE10 self-review and the story tests rather than another frontend review. After this rebase: the eight-role frontend review of the rebased site diff. Its should-fix findings are fixed: the AgentsPageLayout route stub rendered the now always-on Advisor card with no config and no loading state, and the enabled doc described server storage. The failed roles were run again until no should-fix finding remained. A self-check against Danielle's review patterns predicted approval with no comments.
  • Codex: requested on 3f5d0a8 at 16:02 UTC; it returned three threads at 16:08 UTC. Two were fixed in 799508d (a toBeVisible assertion in a story play function, against FE1; a docs sentence split across lines, against DOC1). The third asked for a TODO in coderd/x/chatd/ARCHITECTURE.md. It was first declined under Mike's instruction not to edit that file; Mike then asked for the document to be fixed when a PR makes it wrong, so 251d781 corrects line 448 and the thread has a follow-up reply. All three threads are resolved. The second pass on 799508d found no major issues at 16:20 UTC. The third pass on 251d781 found no major issues at 22:28 UTC. After this rebase, a fourth pass was requested on 6007ee5 and found no major issues at 17:14 UTC. All three threads stay resolved. After the second rebase, a fifth pass was requested on e898d5b and found no major issues at 21:15 UTC.
  • CI: On e898d5b, every check passes or is skipped, including required and Flake Check. In the first attempt, GitHub cancelled seven jobs at 21:19 UTC ("Step canceled by GitHub") and the Windows and Storybook runners lost communication with the server; one rerun of the failed jobs passed, and a rerun of the cancelled Flake Check passed. On 6007ee5, required passes, with 23 checks successful and 6 skipped. The first attempt failed only in test-go-pg (macos-latest), in TestAPISessionRecoveryWriteDeadline/message in agent/x/agentacp, which this PR does not touch and which also fails on main (run). The rerun passed.
  • Final advisory: ready for human review with follow-ups, no blocking findings. Merge risk is high because of a product decision, not code risk: on upgrade every deployment gives root chats the advisor, its guidance block in the system prompt, and no per-turn cap by default (max_uses_per_run 0), bounded only by the step limit. Reviewers should confirm that default. Scripts that call /api/experimental/chats/config/advisor get 404 after upgrade. Its follow-ups are applied in 251d781: the docs label (beta, following the precedent of the Coder Agents graduation, rather than removing the label), both copy strings, and the two "per-run" comments.
  • Deferred minor findings: very large limits are accepted; an empty PUT body resets both limits; error cards show text written for the model, and the error for two parallel advisor calls says "without other tools"; at high reasoning effort with 3000 max output tokens, reasoning can use the whole budget and leave an empty reply. From the frontend review of the rebase: the Advisor card shows 0 and 0 in its disabled fields while the config loads or after a load error, which predates this PR but now reaches every admin; no story covers those two states; CoderAgentsPage has no test for the advisor query gate; splitting the two runtime fixes into their own PR was suggested and not done, because the graduation's UAT found both and they block it.

Xum acted on behalf of Mike.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T21:15:31.733638Z e898d5b Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3f5d0a81d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/chat_routes.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 799508da9c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum acted on behalf of Mike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: 251d781f2a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike marked this pull request as ready for review October 29, 2026 10:09
@ibetitsmike
ibetitsmike requested a review from a team as a code owner October 29, 2026 10:09
@pr-queue-manager
pr-queue-manager Bot requested a review from jakehwll October 29, 2026 10:09
@ibetitsmike
ibetitsmike force-pushed the mike/graduate-advisor branch from 251d781 to 6007ee5 Compare October 29, 2026 17:11
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 6007ee5318

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Remove the chat-advisor experiment. Root chats get the advisor tool
without opt-in, keeping the plan-mode, child-chat, and explore-subagent
restrictions and the configured limits and model overrides.

Move the advisor config routes to /api/v2/chats/config/advisor with
Swagger docs, promote the SDK methods to Client, and drop the
experimental route. The Coder Agents settings page always shows the
advisor controls, and the docs no longer describe an experiment.
chatd rebuilds the advisor runtime for every model step, and the
advisor must be called alone in its step, so max_uses_per_run never
tripped. Seed each runtime with the advisor calls that already consumed
a use since the latest user prompt, so the limit holds across steps and
compaction within a turn and resets on the next user prompt. Error and
limit results do not count, matching the runtime's refund behavior,
while truncated advice still does.

Also hide the rejected deprecated request fields from the generated
PUT /api/v2/chats/config/advisor example, document its 400 and 403
responses, and correct the advisor docs on per-turn limits and tool
exclusivity.
TestChatRouteMounts listed /api/v2/chats/config/advisor as both 200 and 404 after the main merge. The advisor config route is promoted, so keep the 200 assertion and the legacy /api/experimental 404 assertion.
…text

The advisor tool runs in a generation step whose prompt already holds the
parent's advisor call. chatprompt answers that unanswered call with a
synthetic "tool call was interrupted" error, so the nested advisor saw
its own request reported as a failure and sometimes answered that
instead of the question. Drop the call and its placeholder result from
the snapshot passed to the advisor; the question still arrives as the
final user message.
Store enabled=false in the tool-policy fixture so the root-chat case
proves a stored value cannot remove the advisor, assert the organization
settings view renders the Advisor override row, and check the nested
advisor prompt for the advisor's own call note instead of the
placeholder wording.
Omitting the advisor's in-flight call from its context left the user's
request to consult the advisor as the latest turn. On a prompt such as
"call the advisor tool, then report exactly what it returned", the nested
model then acted as the parent: it told the parent to call the advisor
or wrote a report of the call instead of answering.

Replace the call's synthetic interrupted result with a pending result
that says the advisor is answering this call and that earlier
instructions address the parent agent. Replaying the recorded nested
requests against gpt-5-mini, bad answers on that prompt fell from 20 of
20 to 0 of 30 at low reasoning effort and 0 of 20 at high effort, with no
change on neutral questions. One high-effort reply was empty because
reasoning used the whole output budget, which is a separate known issue.
Drop the toBeVisible assertion from the organization settings story's
play function; FE1 leaves rendered-state checks to the Pixel screenshot,
which already captures the Advisor form. Put each sentence of the
advisor docs on its own source line, as DOC1 requires.
The advisor no longer needs an experiment, so its docs nav label moves
from early access, which the feature stages page defines as disabled by
default, to beta. The settings help text said 0 means unlimited for both
limits, but 0 max tokens uses the server default. The limit card said the
limit applies to the conversation, but the quota is per turn.

The chatd architecture document still listed the advisor config routes
as answering only on /api/experimental after they moved to /api/v2.
@ibetitsmike
ibetitsmike force-pushed the mike/graduate-advisor branch from 6007ee5 to e898d5b Compare October 29, 2026 21:13
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: e898d5bf75

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike merged commit 31b8fd1 into main Oct 29, 2026
45 of 56 checks passed
@ibetitsmike
ibetitsmike deleted the mike/graduate-advisor branch October 29, 2026 22:24
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 29, 2026
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

Squash-merged as 31b8fd1 after the rebase onto main 4e8c5bf: Codex found no major issues on e898d5b, required and every other check passed, and the approval stayed in place.

Xum merged this PR on behalf of @ibetitsmike.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants