Repository navigation
feat: graduate MCP tool search from experiments - #31327
Conversation
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
|
@codex review
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review
|
2dd2f70 to
7a05cdf
Compare
|
@codex review
|
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@codex review
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 51e371f6a1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review
|
|
Codex Review: Didn't find any major issues. What shall we delve into next? Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
6d7d208 to
e65a997
Compare
|
@codex review
|
|
Codex Review: Didn't find any major issues. Another round soon, please! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Agent chats now defer MCP tool schemas behind the find_tools catalog without the mcp-tool-search experiment. Every generation with deferrable external or workspace MCP tools defers; generations without candidates, or with a find_tools name collision, keep eager tools. Removes the experiment from the registry and the user-scoped ruleguard list, the per-turn experiment decision cache, and the chatd experiment evaluator dependency. Experiment-rule tests, CLI examples, and docs now use the example experiment.
… parse offeredToolNames ignored a json.Unmarshal error and returned the advertised tools without the find_tools catalog, so a request body that did not parse could go unnoticed. Report it with assert.NoError. The helper runs in the mock OpenAI server's handler goroutines, where require would call t.FailNow off the test goroutine, so it takes the test's testing.TB, records the failure, and still returns the names it collected. Also reword the recordedOpenAIRequest.Offered comment: Offered is Tools plus the deferred MCP tools listed in the find_tools catalog.
… the runner architecture The runner no longer keeps a per-turn mcp-tool-search decision. This PR removes turnExperimentDecisions and the TurnExperiments input, and chatd offers find_tools whenever a step has deferrable MCP tools. Remove the "Local state" bullet and the paragraph that described that cache. #29964 added both, so the section now reads as it did before that change.
…periment rule mcp-tool-search no longer accepts runtime rules, but a rule stored before the upgrade stays in site_configs. The experiment rules API then lists it as ignored, and no supported interface can delete it: PUT and reset return 400 for an experiment that is not user-scoped, and there is no DELETE route. Migration 000637 deletes experiment_rule:mcp-tool-search, as 000616 did for chat-automations. The down migration does not restore the rule. A TestMigrationChain step checks that the upgrade deletes this rule and keeps the rule of another experiment.
e65a997 to
4b32cb7
Compare
|
@codex review
|
|
Codex Review: Didn't find any major issues. Delightful! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Graduates MCP tool search: agent chats now defer MCP tool schemas behind the
find_toolscatalog without themcp-tool-searchexperiment or an experiment rule.find_toolswith a catalog in place of the full schemas. Inline MCP servers stay eager, and afind_toolsname collision still falls back to eager tools. The experiment had no catalog-size threshold, so this PR adds none.mcp-tool-searchfrom the experiment registry, the user-scoped experiment list and its ruleguard check, and the per-turn experiment decisions, along with chatd'sConfig.ExperimentEvaluator.chatd.Newstill returns an error because it validates internal MCP servers.exampleexperiment, and the agents docs drop the opt-in wording.mcp-tool-searchbefore the upgrade. Without it, the rules API would keep listing that rule as ignored and could not clear it: PUT andresetreturn 400, and there is no DELETE route.coderd/x/chatd/ARCHITECTURE.mdno longer describes the runner's per-turnmcp-tool-searchdecision. Mike asked to fix the document in this PR, so it removes that bullet and paragraph from "Local state".Related: #30919 lists
mcp-tool-searchas early access inexperiments/registry.yaml. #30210 adds a user-scoped experiment and changes the samescripts/rules.go,codersdk/deployment.goand rule tests. Whichever PR lands second needs a small rebase.Delivery record
ARCHITECTURE.md, and 4b32cb7 adds migration 000637. On 2026-10-29 Mike asked for another rebase, so the four commits 7a05cdf, 51e371f, a5ae50b and 6d7d208 moved from main dbd7db6 to main 5c691c8. Two files conflicted, both because of main's feat!: choose the computer use model per organization #31300. Incoderd/x/chatd/turn_environment_internal_test.go, feat!: choose the computer use model per organization #31300 replaced the computer use transport test with the table testTestBuildTurnEnvironmentComputerUseRunsOnChatModel, so the rebase keeps main's test and removesTurnExperimentsfrom itsbuildTurnEnvironmentcall. Incoderd/database/migrations/migrate_test.go, feat!: choose the computer use model per organization #31300 added a 000635 chain step. Main now has migrations 000635 (feat!: choose the computer use model per organization #31300) and 000636 (feat: let organization members turn off organization skills in chats #31134), so this PR's migration moved from 000635 to 000637: the file names, the chain step, its test function (now after main's 000635 step function) and the commit message. The rebased head e65a997 also fixed two comments: the migration comment gives the reason in the present tense, and thedecideMCPToolSearchdoc comment drops the sentence "Every generation with deferrable candidates defers", which afind_toolsname collision makes untrue. Two later fixes, folded into the same commits, gave 4b32cb7.mockFetchedExperimentsinsite/src/api/queries/experiments.test.tsxis a real experiment again (["notifications"]):[]duplicated the sharedMockExperimentsand also matched an empty result. Five comments that only repeated the code are deleted: thenewChatdServerdoc, two "should be offered to the LLM" comments in the chatd MCP tests, theWritesApplyToNextRequestsetup comment and the doc comment of the 000637 chain step function. The other 29 changed files add and remove the same lines as on 6d7d208, andmake -B genleft the tree clean. Before that, Mike asked for a first rebase, so the head 2dd2f70 (graduation commit 8227587 plus a merge of main 34ef30e) became 7a05cdf, one commit with the same tree on main dbd7db6. In that rebase, main's test(coderd/x/chatd): fold the effort clamp fixture into the pending-user test #30942 had folded the reasoning-effort clamp test into the pending-user test incoderd/x/chatd/turn_environment_internal_test.go,docs/reference/api/schemas.mdwas regenerated, and the MCP tool search production files stayed byte-identical to 2dd2f70. 51e371f addresses the final advisory's should-fix 2:offeredToolNamesnow fails the test throughassert.NoErrorwhen a request body does not parse, instead of ignoring the error. a5ae50b removes the per-turnmcp-tool-searchdecision fromARCHITECTURE.md, and 6d7d208 added the migration as 000635. Both fix Codex's findings on 51e371f.CODER_HOOK_RUN_ALL=1) passed in 165s. The full pre-push hook (CODER_HOOK_RUN_ALL=1) then passed in 592s:make test(480s),make test-js(80s), and the site build (29s). The agent's background runner starts commands with SIGHUP ignored, and agent/agentscripts and agent/agentssh stop child processes with SIGHUP, so the pre-push run reset SIGHUP to its default, as on 2dd2f70. On 51e371f, the full pre-commit hook passed in 51s and the full pre-push hook in 200s (make test108s,make test-js89s; the site build target was already current), with SIGHUP reset the same way. On a5ae50b and 6d7d208, the full pre-commit hook passed in 56s and 43s, and the full pre-push hook passed in 534s (make test411s,make test-js87s, the site build 33s), with SIGHUP reset the same way. For e65a997, the full pre-commit hook passed on the two fixup commits that carry its comment fixes (199s and 83s). For 4b32cb7, it passed on the three fixup commits of the follow-up fixes (77s, 76s and 77s). Each autosquash left the tree identical to the last fixup's tree, so the full hook, which runsmake gen,make fmtandmake lint, passed on both heads' trees. The pre-push hook was skipped under Mike's standing rule, so CI is the full test run for these heads.decideMCPToolSearchfailsTestDecideMCPToolSearchandTestMCPToolSearchGenerationFlows. On 51e371f,go vetandgo test ./coderd/x/chatd/pass. As a negative control, a corrupted request body makesTestPlanTurnPromptContractfail with the new assert message, while the old helper let the same test pass. On 6d7d208,TestMigrationChainandgo test ./coderd/database/migrations/pass. The new chain step fails when the migration'sDELETEis removed, and when the delete also removes another experiment's rule. On e65a997,go build ./...,go vetof the touched packages,go test ./coderd/x/chatd/,go test ./coderd/database/migrations/(with the 000637TestMigrationChainstep), the coderd, enterprise, CLI and codersdk experiment tests, and the experiments Vitest test pass. On 4b32cb7,go vetof the chatd, migrations and coderd packages,go test ./coderd/x/chatd/, the 000637TestMigrationChainstep, the coderd experiment tests and the experiments Vitest test pass. With["notifications"], a query function that drops the fetched list fails all three experiments tests; with[], the same change passed all three. The chain step's logic did not change, so its red-green proof on 6d7d208 still applies.wfr_child_p0Vt5MpuzeTAgZZ0ojL_vyfLO5-V7ym5passed validation (13 checks), UAT and the test audit. It still reportedreadyForPr=falseon one gate: the auditor made no commit and recorded prose inauditedTreein place of a tree SHA.auditedTree, fromgit rev-parse 2dd2f70e81a^{tree}(=b1f2cdaf), and checked all 18 packaged gates: scope, newTests evidence, tree, raw exit codes, and repeated UAT (not required, because the audit changed no files). A separate record now shows recoveredreadyForPr=true. The original run still showsreadyForPr=false.decideMCPToolSearchdoc comment and the migration's number and comment, their production hunks are the same as on 6d7d208.find_toolssearch, activation and invocation work within a turn and across turns, after a page reload, and after a coderd restart.model_intentlabel.experiments.test.tsx:17) the non-empty fixture fixes, with no other findings; a self-check with thedanielle-frontend-reviewskill on the whole PR at 4b32cb7, which predicts approval with no comments (its first pass found the five comments above, now deleted); and Codex on 4b32cb7 (no major issues).@codex reviewwas requested on 4b32cb7 at 17:37 UTC on 2026-10-29. Codex finished at 17:39 UTC with no major issues and opened no review threads. Before that,@codex reviewwas requested on e65a997 at 16:41 UTC on 2026-10-29. Codex finished at 16:44 UTC with no major issues and opened no review threads. Before that,@codex reviewwas requested on 6d7d208 at 22:41 UTC on 2026-10-28. Codex finished at 22:43 UTC with no major issues and opened no review threads. Before that,@codex reviewwas requested on 51e371f at 21:32 UTC on 2026-10-28, and Codex reviewed it at 21:35 UTC with two findings. The P1 on theARCHITECTURE.mdtext was first declined, then fixed in a5ae50b at Mike's request. The P2 on the stored rule is fixed in 6d7d208. Both threads are resolved. Before that,@codex reviewwas requested on 7a05cdf at 13:56 UTC on 2026-10-28. Codex finished at 13:59 UTC with no major issues. The two requests on 2dd2f70 (05:46 UTC and 06:33 UTC on 2026-10-27) got no verdict: the review summary stayed at Running, and Codex posted no review or threads.requiredpassed at 19:06 UTC, and every other check passed or was skipped. The firsttest-go-pg (macos-latest)attempt sat queued from 17:38 UTC while a newer job with the same runner label ran, so the run was cancelled at 17:55 UTC and the macOS job was rerun once. The rerun waited for a runner in a repository-wide macOS backlog, started at 18:58 UTC and passed. On e65a997,test-go-race-pgfailed inagent/x/agentacpTestAPISessionRecoveryWriteDeadline/message, a package this PR does not change: the test failed because slogtest treats an ACP SDK error log ("failed to handle notification ... ACP harness unavailable") as a test failure. Under the race detector it failed locally in 1 of 340 runs on this branch, and in 1 of 340 with main'scodersdk/deployment.go, the only file this PR changes in that package's dependencies, so the PR does not cause it. Main's own CI failed the same subtest intest-go-pg-17on ea6267e (job). The push of 4b32cb7 replaced that run. Before the rebase, 6d7d208 was green: itsrequiredcheck passed at 01:46 UTC on 2026-10-29, after a rerun of atest-go-pg (macos-latest)job that sat queued while newer jobs with the same runner label ran.json.Unmarshalerror inofferedToolNamesis fixed in 51e371f. Deployments that still setCODER_EXPERIMENTS=mcp-tool-searchlog an "ignoring unknown experiment" warning while the feature stays on, which is worth a release note.