Repository navigation
feat: graduate the chat advisor from experiments - #31414
Conversation
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3f5d0a81d8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. 🎉 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review
|
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
251d781 to
6007ee5
Compare
|
@codex review
|
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Remove the chat-advisor experiment. Root chats get the advisor tool without opt-in, keeping the plan-mode, child-chat, and explore-subagent restrictions and the configured limits and model overrides. Move the advisor config routes to /api/v2/chats/config/advisor with Swagger docs, promote the SDK methods to Client, and drop the experimental route. The Coder Agents settings page always shows the advisor controls, and the docs no longer describe an experiment.
chatd rebuilds the advisor runtime for every model step, and the advisor must be called alone in its step, so max_uses_per_run never tripped. Seed each runtime with the advisor calls that already consumed a use since the latest user prompt, so the limit holds across steps and compaction within a turn and resets on the next user prompt. Error and limit results do not count, matching the runtime's refund behavior, while truncated advice still does. Also hide the rejected deprecated request fields from the generated PUT /api/v2/chats/config/advisor example, document its 400 and 403 responses, and correct the advisor docs on per-turn limits and tool exclusivity.
TestChatRouteMounts listed /api/v2/chats/config/advisor as both 200 and 404 after the main merge. The advisor config route is promoted, so keep the 200 assertion and the legacy /api/experimental 404 assertion.
…text The advisor tool runs in a generation step whose prompt already holds the parent's advisor call. chatprompt answers that unanswered call with a synthetic "tool call was interrupted" error, so the nested advisor saw its own request reported as a failure and sometimes answered that instead of the question. Drop the call and its placeholder result from the snapshot passed to the advisor; the question still arrives as the final user message.
Store enabled=false in the tool-policy fixture so the root-chat case proves a stored value cannot remove the advisor, assert the organization settings view renders the Advisor override row, and check the nested advisor prompt for the advisor's own call note instead of the placeholder wording.
Omitting the advisor's in-flight call from its context left the user's request to consult the advisor as the latest turn. On a prompt such as "call the advisor tool, then report exactly what it returned", the nested model then acted as the parent: it told the parent to call the advisor or wrote a report of the call instead of answering. Replace the call's synthetic interrupted result with a pending result that says the advisor is answering this call and that earlier instructions address the parent agent. Replaying the recorded nested requests against gpt-5-mini, bad answers on that prompt fell from 20 of 20 to 0 of 30 at low reasoning effort and 0 of 20 at high effort, with no change on neutral questions. One high-effort reply was empty because reasoning used the whole output budget, which is a separate known issue.
Drop the toBeVisible assertion from the organization settings story's play function; FE1 leaves rendered-state checks to the Pixel screenshot, which already captures the Advisor form. Put each sentence of the advisor docs on its own source line, as DOC1 requires.
The advisor no longer needs an experiment, so its docs nav label moves from early access, which the feature stages page defines as disabled by default, to beta. The settings help text said 0 means unlimited for both limits, but 0 max tokens uses the server default. The limit card said the limit applies to the conversation, but the quota is per turn. The chatd architecture document still listed the advisor config routes as answering only on /api/experimental after they moved to /api/v2.
6007ee5 to
e898d5b
Compare
|
@codex review
|
|
Codex Review: Didn't find any major issues. Hooray! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Squash-merged as 31b8fd1 after the rebase onto main 4e8c5bf: Codex found no major issues on e898d5b,
|
Graduates the chat advisor: root chats get the
advisortool without thechat-advisorexperiment. Plan-mode turns, child chats and explore subagents still do not get it, and the configured limits and advisor model override still apply./api/experimental/chats/config/advisorto/api/v2/chats/config/advisorwith Swagger docs, and promotes the SDK methods toClient.enabledis alwaystrueand read-only. The experimental/chats/configsegment stays reserved, so the old paths return 404 rather than reaching the{chat}route and failing UUID parsing with a 400.chat-advisorfrom the experiment registry. The Coder Agents settings page always shows the advisor controls, and the docs drop the opt-in wording and label the Advisor page beta instead of early access, which the feature stages page defines as disabled by default.max_uses_per_run, which reset on every model step because chatd rebuilds the advisor runtime per step. The runtime now counts successful advisor results since the latest user prompt, including across a compaction inside the turn. Provider errors do not use up the limit.coderd/x/chatd/ARCHITECTURE.mdline 448, which still listed theadvisorroutes as experimental-only.Size: +769 and -223 lines. Generated API docs account for 280 net lines: as public
/api/v2endpoints, the two advisor routes need Swagger annotations, which experimental routes do not have. Apart from those, the graduation commit removes 81 hand-written lines. The rest is the two fixes: the use limit fix adds 50 lines of code and 171 of tests, and the pending call fix adds 51 lines of code and 77 of tests.Delivery record
TestBuildTurnEnvironmentToolPolicy. The advisor routes were then the last routes in the experimental/chats/configgroup, so the graduation commit now reserves that segment, matching the reserved segments in the/api/v2chats routes; without it,TestChatRouteMountsfailed with a 400 forGET /api/experimental/chats/config/system-prompt. Main'sTestChatModelOverridesComputerUseRequiresVirtualDesktopExperimentused the removedExperimentChatAdvisoras an unrelated experiment and now usesExperimentAgentLifecycleHooks. Review cleanups squashed into their commits: rewrapped comments, a corrected test comment, a shorterenabledfield doc, and a sharedMockAdvisorConfigstory fixture. Commits named below by their earlier SHAs are now: 15b689c is 8a4462a (graduation), 9dabee9 is db19b90 (per-turn limit), 72d1024 is 641ad8d (route assertion), 53dbb5c is bfba521 (first context fix), 27053d9 is d1b3d3c (test audit fixes), 3f5d0a8 is 348c390 (pending note), 799508d is c095b13 (Codex fixes) and 251d781 is e898d5b (final advisory follow-ups). A second rebase moved the branch from main 3184d73 to main 4e8c5bf, after feat: graduate MCP tool search from experiments #31327 graduated MCP tool search. Its only conflict was the generated experiment list indocs/reference/api/schemas.md, which now drops both flags;git range-diffshows every other patch unchanged.CODER_HOOK_RUN_ALL=1, SIGHUP reset to default) on each push and passed; on 3f5d0a8:make test(325s) andmake test-js(82s); on 799508d:make test(102s),make test-js(81s) and the site build (29s); on 251d781:make test(198s),make test-js(94s) and the site build (35s). After this rebase, the full pre-commit hook passed on each fixup commit; under Mike's standing rule the pre-push hook was not run for the 6007ee5 and e898d5b pushes, and CI covers them.go test ./coderd/x/chatd/...and the focused coderd, telemetry and codersdk advisor, route and experiment tests pass.TestAdvisorToolMarksItsOwnPendingCallpins the exact nested prompt, andTestAdvisorHappyPath_RootChatchecks the real prompt pipeline for the pending note and for no interrupted result. Both fail against 27053d9 (call dropped), and the end-to-end check also fails against 72d1024 (interrupted result). On the rebased branch:go build ./..., compiling the coderd, codersdk and enterprise tests,go test ./coderd/x/chatd/..., the focused coderd route, advisor config and computer-use tests, and the telemetry and codersdk tests pass; on the site,tsc, Biome, the API andOrganizationAgentSettingsVitest suites (392 tests) and the changed stories plus the advisor tool stories (326 tests) pass. After the last change, which only moved the story fixture,tsc, Biome and the two affected story files (37 tests) pass. After the second rebase,make genleaves the tree clean, andgo build ./..., the coderd, codersdk and enterprise test builds,go test ./coderd/x/chatd/..., the focused coderd, telemetry and codersdk tests,tsc, Biome, oxlint, 392 Vitest tests and 84 story tests pass.AgentsPageLayoutroute stub rendered the now always-on Advisor card with no config and no loading state, and theenableddoc described server storage. The failed roles were run again until no should-fix finding remained. A self-check against Danielle's review patterns predicted approval with no comments.toBeVisibleassertion in a storyplayfunction, against FE1; a docs sentence split across lines, against DOC1). The third asked for a TODO incoderd/x/chatd/ARCHITECTURE.md. It was first declined under Mike's instruction not to edit that file; Mike then asked for the document to be fixed when a PR makes it wrong, so 251d781 corrects line 448 and the thread has a follow-up reply. All three threads are resolved. The second pass on 799508d found no major issues at 16:20 UTC. The third pass on 251d781 found no major issues at 22:28 UTC. After this rebase, a fourth pass was requested on 6007ee5 and found no major issues at 17:14 UTC. All three threads stay resolved. After the second rebase, a fifth pass was requested on e898d5b and found no major issues at 21:15 UTC.requiredand Flake Check. In the first attempt, GitHub cancelled seven jobs at 21:19 UTC ("Step canceled by GitHub") and the Windows and Storybook runners lost communication with the server; one rerun of the failed jobs passed, and a rerun of the cancelled Flake Check passed. On 6007ee5,requiredpasses, with 23 checks successful and 6 skipped. The first attempt failed only intest-go-pg (macos-latest), inTestAPISessionRecoveryWriteDeadline/messageinagent/x/agentacp, which this PR does not touch and which also fails on main (run). The rerun passed.max_uses_per_run0), bounded only by the step limit. Reviewers should confirm that default. Scripts that call/api/experimental/chats/config/advisorget 404 after upgrade. Its follow-ups are applied in 251d781: the docs label (beta, following the precedent of the Coder Agents graduation, rather than removing the label), both copy strings, and the two "per-run" comments.CoderAgentsPagehas no test for the advisor query gate; splitting the two runtime fixes into their own PR was suggested and not done, because the graduation's UAT found both and they block it.