Skip to content

feat: graduate MCP tool search from experiments - #31327

Merged
ibetitsmike merged 4 commits into
mainfrom
mike/graduate-mcp-tool-search
Oct 29, 2026
Merged

ibetitsmike merged 4 commits into
mainfrom
mike/graduate-mcp-tool-search

Conversation

@ibetitsmike

@ibetitsmike ibetitsmike commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

Graduates MCP tool search: agent chats now defer MCP tool schemas behind the find_tools catalog without the mcp-tool-search experiment or an experiment rule.

  • Every generation with deferrable MCP tools (MCP server configs and workspace MCP tools) offers find_tools with a catalog in place of the full schemas. Inline MCP servers stay eager, and a find_tools name collision still falls back to eager tools. The experiment had no catalog-size threshold, so this PR adds none.
  • Removes mcp-tool-search from the experiment registry, the user-scoped experiment list and its ruleguard check, and the per-turn experiment decisions, along with chatd's Config.ExperimentEvaluator. chatd.New still returns an error because it validates internal MCP servers.
  • The experiment-rules tests, CLI examples and docs now use the example experiment, and the agents docs drop the opt-in wording.
  • Migration 000637 deletes a rule stored for mcp-tool-search before the upgrade. Without it, the rules API would keep listing that rule as ignored and could not clear it: PUT and reset return 400, and there is no DELETE route.
  • coderd/x/chatd/ARCHITECTURE.md no longer describes the runner's per-turn mcp-tool-search decision. Mike asked to fix the document in this PR, so it removes that bullet and paragraph from "Local state".

Related: #30919 lists mcp-tool-search as early access in experiments/registry.yaml. #30210 adds a user-scoped experiment and changes the same scripts/rules.go, codersdk/deployment.go and rule tests. Whichever PR lands second needs a small rebase.

Delivery record

  • History: four commits on main 5c691c8. ff5f48b is the graduation, 906de03 is test-only, f2b8ee0 updates ARCHITECTURE.md, and 4b32cb7 adds migration 000637. On 2026-10-29 Mike asked for another rebase, so the four commits 7a05cdf, 51e371f, a5ae50b and 6d7d208 moved from main dbd7db6 to main 5c691c8. Two files conflicted, both because of main's feat!: choose the computer use model per organization #31300. In coderd/x/chatd/turn_environment_internal_test.go, feat!: choose the computer use model per organization #31300 replaced the computer use transport test with the table test TestBuildTurnEnvironmentComputerUseRunsOnChatModel, so the rebase keeps main's test and removes TurnExperiments from its buildTurnEnvironment call. In coderd/database/migrations/migrate_test.go, feat!: choose the computer use model per organization #31300 added a 000635 chain step. Main now has migrations 000635 (feat!: choose the computer use model per organization #31300) and 000636 (feat: let organization members turn off organization skills in chats #31134), so this PR's migration moved from 000635 to 000637: the file names, the chain step, its test function (now after main's 000635 step function) and the commit message. The rebased head e65a997 also fixed two comments: the migration comment gives the reason in the present tense, and the decideMCPToolSearch doc comment drops the sentence "Every generation with deferrable candidates defers", which a find_tools name collision makes untrue. Two later fixes, folded into the same commits, gave 4b32cb7. mockFetchedExperiments in site/src/api/queries/experiments.test.tsx is a real experiment again (["notifications"]): [] duplicated the shared MockExperiments and also matched an empty result. Five comments that only repeated the code are deleted: the newChatdServer doc, two "should be offered to the LLM" comments in the chatd MCP tests, the WritesApplyToNextRequest setup comment and the doc comment of the 000637 chain step function. The other 29 changed files add and remove the same lines as on 6d7d208, and make -B gen left the tree clean. Before that, Mike asked for a first rebase, so the head 2dd2f70 (graduation commit 8227587 plus a merge of main 34ef30e) became 7a05cdf, one commit with the same tree on main dbd7db6. In that rebase, main's test(coderd/x/chatd): fold the effort clamp fixture into the pending-user test #30942 had folded the reasoning-effort clamp test into the pending-user test in coderd/x/chatd/turn_environment_internal_test.go, docs/reference/api/schemas.md was regenerated, and the MCP tool search production files stayed byte-identical to 2dd2f70. 51e371f addresses the final advisory's should-fix 2: offeredToolNames now fails the test through assert.NoError when a request body does not parse, instead of ignoring the error. a5ae50b removes the per-turn mcp-tool-search decision from ARCHITECTURE.md, and 6d7d208 added the migration as 000635. Both fix Codex's findings on 51e371f.
  • Hooks: on 7a05cdf, the full pre-commit hook (CODER_HOOK_RUN_ALL=1) passed in 165s. The full pre-push hook (CODER_HOOK_RUN_ALL=1) then passed in 592s: make test (480s), make test-js (80s), and the site build (29s). The agent's background runner starts commands with SIGHUP ignored, and agent/agentscripts and agent/agentssh stop child processes with SIGHUP, so the pre-push run reset SIGHUP to its default, as on 2dd2f70. On 51e371f, the full pre-commit hook passed in 51s and the full pre-push hook in 200s (make test 108s, make test-js 89s; the site build target was already current), with SIGHUP reset the same way. On a5ae50b and 6d7d208, the full pre-commit hook passed in 56s and 43s, and the full pre-push hook passed in 534s (make test 411s, make test-js 87s, the site build 33s), with SIGHUP reset the same way. For e65a997, the full pre-commit hook passed on the two fixup commits that carry its comment fixes (199s and 83s). For 4b32cb7, it passed on the three fixup commits of the follow-up fixes (77s, 76s and 77s). Each autosquash left the tree identical to the last fixup's tree, so the full hook, which runs make gen, make fmt and make lint, passed on both heads' trees. The pre-push hook was skipped under Mike's standing rule, so CI is the full test run for these heads.
  • Tests: on 7a05cdf, the chatd suite, the targeted coderd, enterprise, CLI and codersdk experiment tests, and the experiments Vitest test pass. On 2dd2f70, the test audit broke 10 production behaviors to prove that the changed tests fail. For example, putting the experiment gate back in decideMCPToolSearch fails TestDecideMCPToolSearch and TestMCPToolSearchGenerationFlows. On 51e371f, go vet and go test ./coderd/x/chatd/ pass. As a negative control, a corrupted request body makes TestPlanTurnPromptContract fail with the new assert message, while the old helper let the same test pass. On 6d7d208, TestMigrationChain and go test ./coderd/database/migrations/ pass. The new chain step fails when the migration's DELETE is removed, and when the delete also removes another experiment's rule. On e65a997, go build ./..., go vet of the touched packages, go test ./coderd/x/chatd/, go test ./coderd/database/migrations/ (with the 000637 TestMigrationChain step), the coderd, enterprise, CLI and codersdk experiment tests, and the experiments Vitest test pass. On 4b32cb7, go vet of the chatd, migrations and coderd packages, go test ./coderd/x/chatd/, the 000637 TestMigrationChain step, the coderd experiment tests and the experiments Vitest test pass. With ["notifications"], a query function that drops the fetched list fails all three experiments tests; with [], the same change passed all three. The chain step's logic did not change, so its red-green proof on 6d7d208 still applies.
  • Workflow (on 2dd2f70): implement-flow, changed only to accept a verified no-op implement stage, because the implementation was already committed. Run wfr_child_p0Vt5MpuzeTAgZZ0ojL_vyfLO5-V7ym5 passed validation (13 checks), UAT and the test audit. It still reported readyForPr=false on one gate: the auditor made no commit and recorded prose in auditedTree in place of a tree SHA.
  • Recovery receipt (on 2dd2f70): a deterministic receipt replayed the recorded steps through the packaged workflow code with the installed runtime's own compiler and step hashing. It derived only auditedTree, from git rev-parse 2dd2f70e81a^{tree} (= b1f2cdaf), and checked all 18 packaged gates: scope, newTests evidence, tree, raw exit codes, and repeated UAT (not required, because the audit changed no files). A separate record now shows recovered readyForPr=true. The original run still shows readyForPr=false.
  • Remote UAT through Coder Agents on 2dd2f70: https://dogfood.cdr.dev/agents/cdc4fdb1-193c-471c-9d35-5a0874381d7e. The isolation proof and leak audits passed. The chat used a real model with no experiment opt-in. UAT was not repeated on 7a05cdf, because the rebase left the MCP tool search production files byte-identical. It was not repeated on e65a997 or 4b32cb7 either: apart from the decideMCPToolSearch doc comment and the migration's number and comment, their production hunks are the same as on 6d7d208.
    • The retired experiment is gone from discovery, and the rules API rejects it.
    • find_tools search, activation and invocation work within a turn and across turns, after a page reload, and after a coderd restart.
    • Chats with no MCP servers keep eager built-in tools.
    • Six minor findings, none blocking: search can activate every tool on a server whose description matches the query; the stored-rule case above (fixed by migration 000637); a vague error after a server is disabled; two docs gaps; direct catalog calls skip the model_intent label.
  • Reviews (7 of 7 completed before the 2026-10-29 rebase; Mike raised the cap to fix the architecture document): consolidated frontend review on 8227587 with no findings (neither the merge nor the rebase changes the site diff), UAT endorsement (pass), test audit (pass, no changes), Codex on 7a05cdf (no major issues), a fresh final advisory on 7a05cdf (ready for human review with follow-ups, merge risk medium, no blocking findings; it predates 51e371f, a5ae50b and 6d7d208), Codex on 51e371f (two findings, both fixed in a5ae50b and 6d7d208), and Codex on 6d7d208 (no major issues). After the rebase: Codex on e65a997 (no major issues); a frontend review of the site diff on e65a997 (8 roles), whose only should-fix finding (TS3 on experiments.test.tsx:17) the non-empty fixture fixes, with no other findings; a self-check with the danielle-frontend-review skill on the whole PR at 4b32cb7, which predicts approval with no comments (its first pass found the five comments above, now deleted); and Codex on 4b32cb7 (no major issues).
  • Codex: @codex review was requested on 4b32cb7 at 17:37 UTC on 2026-10-29. Codex finished at 17:39 UTC with no major issues and opened no review threads. Before that, @codex review was requested on e65a997 at 16:41 UTC on 2026-10-29. Codex finished at 16:44 UTC with no major issues and opened no review threads. Before that, @codex review was requested on 6d7d208 at 22:41 UTC on 2026-10-28. Codex finished at 22:43 UTC with no major issues and opened no review threads. Before that, @codex review was requested on 51e371f at 21:32 UTC on 2026-10-28, and Codex reviewed it at 21:35 UTC with two findings. The P1 on the ARCHITECTURE.md text was first declined, then fixed in a5ae50b at Mike's request. The P2 on the stored rule is fixed in 6d7d208. Both threads are resolved. Before that, @codex review was requested on 7a05cdf at 13:56 UTC on 2026-10-28. Codex finished at 13:59 UTC with no major issues. The two requests on 2dd2f70 (05:46 UTC and 06:33 UTC on 2026-10-27) got no verdict: the review summary stayed at Running, and Codex posted no review or threads.
  • CI: On 4b32cb7, required passed at 19:06 UTC, and every other check passed or was skipped. The first test-go-pg (macos-latest) attempt sat queued from 17:38 UTC while a newer job with the same runner label ran, so the run was cancelled at 17:55 UTC and the macOS job was rerun once. The rerun waited for a runner in a repository-wide macOS backlog, started at 18:58 UTC and passed. On e65a997, test-go-race-pg failed in agent/x/agentacp TestAPISessionRecoveryWriteDeadline/message, a package this PR does not change: the test failed because slogtest treats an ACP SDK error log ("failed to handle notification ... ACP harness unavailable") as a test failure. Under the race detector it failed locally in 1 of 340 runs on this branch, and in 1 of 340 with main's codersdk/deployment.go, the only file this PR changes in that package's dependencies, so the PR does not cause it. Main's own CI failed the same subtest in test-go-pg-17 on ea6267e (job). The push of 4b32cb7 replaced that run. Before the rebase, 6d7d208 was green: its required check passed at 01:46 UTC on 2026-10-29, after a rerun of a test-go-pg (macos-latest) job that sat queued while newer jobs with the same runner label ran.
  • Final advisory follow-ups: the swallowed json.Unmarshal error in offeredToolNames is fixed in 51e371f. Deployments that still set CODER_EXPERIMENTS=mcp-tool-search log an "ignoring unknown experiment" warning while the feature stays on, which is worth a release note.

Xum acted on behalf of Mike.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T17:39:52.353669Z 4b32cb7 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@ibetitsmike
ibetitsmike force-pushed the mike/graduate-mcp-tool-search branch from 2dd2f70 to 7a05cdf Compare October 28, 2026 13:55
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 7a05cdf0ac

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 51e371f6a1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread coderd/x/chatd/turn_environment.go Resolved
Comment thread codersdk/deployment.go Resolved
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 6d7d208bc3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike marked this pull request as ready for review October 29, 2026 10:09
@pr-queue-manager
pr-queue-manager Bot requested a review from ThomasK33 October 29, 2026 10:09
@ibetitsmike
ibetitsmike force-pushed the mike/graduate-mcp-tool-search branch from 6d7d208 to e65a997 Compare October 29, 2026 16:41
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: e65a997e4e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Agent chats now defer MCP tool schemas behind the find_tools catalog
without the mcp-tool-search experiment. Every generation with
deferrable external or workspace MCP tools defers; generations without
candidates, or with a find_tools name collision, keep eager tools.

Removes the experiment from the registry and the user-scoped ruleguard
list, the per-turn experiment decision cache, and the chatd experiment
evaluator dependency. Experiment-rule tests, CLI examples, and docs now
use the example experiment.
… parse

offeredToolNames ignored a json.Unmarshal error and returned the
advertised tools without the find_tools catalog, so a request body that
did not parse could go unnoticed. Report it with assert.NoError. The
helper runs in the mock OpenAI server's handler goroutines, where
require would call t.FailNow off the test goroutine, so it takes the
test's testing.TB, records the failure, and still returns the names it
collected.

Also reword the recordedOpenAIRequest.Offered comment: Offered is Tools
plus the deferred MCP tools listed in the find_tools catalog.
… the runner architecture

The runner no longer keeps a per-turn mcp-tool-search decision. This PR
removes turnExperimentDecisions and the TurnExperiments input, and chatd
offers find_tools whenever a step has deferrable MCP tools.

Remove the "Local state" bullet and the paragraph that described that
cache. #29964 added both, so the section now reads as it did before
that change.
…periment rule

mcp-tool-search no longer accepts runtime rules, but a rule stored
before the upgrade stays in site_configs. The experiment rules API then
lists it as ignored, and no supported interface can delete it: PUT and
reset return 400 for an experiment that is not user-scoped, and there is
no DELETE route.

Migration 000637 deletes experiment_rule:mcp-tool-search, as 000616 did
for chat-automations. The down migration does not restore the rule. A
TestMigrationChain step checks that the upgrade deletes this rule and
keeps the rule of another experiment.
@ibetitsmike
ibetitsmike force-pushed the mike/graduate-mcp-tool-search branch from e65a997 to 4b32cb7 Compare October 29, 2026 17:37
@ibetitsmike

Copy link
Copy Markdown
Collaborator Author

@codex review

Xum posted this comment on behalf of @ibetitsmike.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 4b32cb7799

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ibetitsmike
ibetitsmike merged commit 1acacc9 into main Oct 29, 2026
56 of 59 checks passed
@ibetitsmike
ibetitsmike deleted the mike/graduate-mcp-tool-search branch October 29, 2026 19:33
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 29, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants