Repository navigation
feat: validate documented experiment keys (DOCS-139) - #29316
nickvigilante merged 7 commits into
Conversation
|
/coder-agents-review |
|
Chat: Review in progress (2/2 reviewers complete) | View chat deep-review v0.9.0 | Round 2 | Last posted: Round 2, 6 findings (2 P2, 1 P3, 1 Nit, 2 Note), COMMENT. Review Finding inventoryFinding inventory: PR #29316Findings
Contested and acknowledgedCRF-5 (Note, Makefile:797) - lint wiring vs #29268 merge order
Round logRound 1Netero-only first-pass gate. P2 present (title scope, matches failed Round 2Churn guard PROCEED: 4 addressed (CRF-1,2,3,4 via a2b07d2), 1 acknowledged (CRF-5). First panel round. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
First-pass review only. These are mechanical findings from Netero; the full review panel has not yet reviewed this PR. The panel will review once these are addressed.
The design is well-argued: a one-way drift guard that validates keys the docs already name, deliberately scoped to explicit enablement forms so it cannot produce the false positives that get a check deleted. Test coverage of the core matching logic is real, and the DisplayName fix is backed by an assertion that would fail against the pre-PR default title-casing.
One P2 blocks the mechanical floor: the PR title scope scripts/checkexperimentkeys does not contain every changed file (Makefile, codersdk/), which is why the title CI check is red. Beyond that: one vacuous test assertion, one stale-Go loop capture, and two notes worth a human decision (coverage of the walk/error paths, and the merge-order coupling with #29268 that can turn main's lint red).
As Netero put it: the loop assertion "as written is vacuous; if the intent is to force an explicit case per experiment, it must assert the value differs from the default title-casing."
Severity count: 1 P2, 1 P3, 1 Nit, 2 Note.
🤖 This review was automatically generated with Coder Agents.
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review, still first-pass only. The full panel has not yet reviewed this PR; a new P2 in the core matcher keeps the mechanical floor unclean, so the panel is deferred one more round and will review once this is addressed.
Round-1 findings resolved and verified: the PR title now omits the scope for a cross-cutting change (CRF-1, title CI green); the vacuous loop assertion is replaced by an internal test that maps display names explicitly and requires one entry per known experiment, rejecting extras (CRF-2); the unnecessary loop capture is gone (CRF-3); and coverage of the walk, dedup, and both exit-2 I/O paths is added, bringing the package to 87% (CRF-4). Good, targeted fixes.
CRF-5 (wiring into make lint turns main red until #29268 merges) remains acknowledged with no ticket. The author's plan is sound (keep the PR draft, target main, wait for #29268), but the merge-order dependency is a human decision: do not merge this before #29268. Not closed.
New this round: one P2. The recognition regex includes , in its character class, so a value followed by ordinary comma punctuation (CODER_EXPERIMENTS=notifications, then restart.) splits into a trailing empty key and reports unknown experiment key "". Reproduced against a one-line Markdown file. As Netero put it, this is "exactly the false-positive failure mode the design states it avoids."
Severity count: 1 P2 (new). CRF-1..4 verified fixed; CRF-5 acknowledged.
🤖 This review was automatically generated with Coder Agents.
bbeeb1b to
7b57abe
Compare
|
/coder-agents-review |
…key lint into make lint The deferred wiring waited on #29268, which merged and removed the three stale documented oauth2 keys. The check is green against main, so the standalone target joins aggregate lint. Keep DisplayName as a switch, matching main, and pin every known experiment's display name from an expected map in the package test so a new experiment cannot silently fall back to a title-cased key.
The note's CODER_EXPERIMENTS=oauth2 example is an explicit enablement form, so lint/check-experiment-keys reports the removed experiment as an unknown key. Describe the removal without naming it as a value.
ExperimentsKnown gained chat-projects, chat-inline-mcp-servers, enable-ai-workspace-debug, chat-board, and chat-stage-metrics on main, so TestExperimentDisplayNames needs their expected names.
7b57abe to
3223e0b
Compare
Docs previewCheck off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here. |
Summary
The docs name specific experiment keys wherever an individual experimental feature is documented, and nothing verifies those keys still exist in the code.
When an experiment is renamed or graduated, the enablement example keeps telling operators to set a key the server no longer accepts.
This adds
scripts/checkexperimentkeys, which fails when a key named indocs/**is absent fromcodersdk.ExperimentsKnown, and wires it intomake lintaslint/check-experiment-keys.The data flows one way.
The check reads keys the docs already name and validates them against code.
It does not generate a page, add a manifest route, or write any key into
docs/.That is deliberate: #29113 retired the generated feature-stage lists because the team decided not to advertise which features are currently experimental or beta.
A guard against stale keys is compatible with that decision; an index is not.
The five key references the check currently guards are the per-feature ones that decision left in place:
chat-advisor,chat-virtual-desktop,auto-fill-parameters,agent-lifecycle-hooks, andmcp-server-http.Recognition rules
Only explicit enablement forms are matched:
--experiments=<key-list>andCODER_EXPERIMENTS=<key-list>, including every key in a comma-separated list.Values may be single- or double-quoted.
Explicit shell line continuations are joined before matching, preserving the first physical line for a finding.
Bare identifiers in prose are ignored on purpose.
A check that guesses at prose produces false positives, and a check with false positives gets deleted by the first person it blocks.
*,feature1,feature2, and angle-bracketed values are recognized placeholders.The recognized forms are documented in the script's README.
Known gap: the space-separated
--experiments <key>form is not matched.No prose uses it today, so this is latent rather than broken, and closing it is a small follow-up if the form reappears.
Also fixed
ExperimentMCPToolSearchhad no case inDisplayName(), so it rendered as a title-cased guess in the product UI as well as anywhere else display names surface.TestExperimentDisplayNamesnow pins the display name of every known experiment against an expected map, with a length assertion, so a new experiment cannot silently fall back to the title-cased key.Verification
go test ./scripts/checkexperimentkeys ./codersdkpasses.Coverage includes known and unknown keys, quoted values, shell continuations, placeholders, multi-key lists, bare-prose exclusion, and failure-output formatting.
golangci-lint run ./scripts/checkexperimentkeys/... ./codersdk/...reports 0 issues.make lint/check-experiment-keysis clean against currentdocs/.CODER_EXPERIMENTS=not-a-real-keyexits 1 with the expected finding.pre-commithook passes, includinglint/check-experiment-keysinside aggregatemake lint.History: the deferred lint wiring, now resolved
This PR previously landed the tool and a standalone
maketarget while leaving the one-line addition to aggregatemake lintfor a follow-up, and stayed draft, because three documentedoauth2keys were stale and owned by another PR:docs/admin/integrations/oauth2-provider.md:34docs/ai-coder/mcp-server.md:108docs/ai-coder/mcp-server.md:114#29268 merged on 2026-09-15 and removed all three in favor of
CODER_OAUTH2_PROVIDER_ENABLE.The branch is rebased onto that, the check is green against
main, the wiring is folded into this PR, and the merge-order constraint recorded as CRF-5 no longer applies.The rebase also collided with
DisplayName(), which gained new cases onmain.Resolved in favor of
main's switch rather than this branch's earlier map refactor, keeping the production diff to the one missing case and moving the exhaustiveness expectations into the test.Why this replaced a generated reference page
DOCS-139 originally asked for a generated
docs/reference/experiments.mdlisting every key inExperimentsKnown.That was built and opened as #29311, then closed without merging.
Commit b23d73d (#29113, six days earlier) had removed both generated feature-stage indexes from
docs/install/releases/feature-stages.md, recording that the decision, after discussion with David, was not to advertise either list, and describing them as "drift surfaces that promise a completeness the docs cannot maintain."It explicitly rejected the superseded #28949, which generated the list from
ExperimentsKnownwith descriptions parsed from Go constant comments, for making docs track "an internal, unstable surface."#29311 was that rejected design rebuilt.
The ticket predated the decision; the page should not have been built on top of it.
This PR is the part of the original goal that survives: the drift protection, without the publication.
Local pre-push note
The pre-push hook's test stage cannot run in this workspace: GNU Make 4.4.1 segfaults on the repo's
.SHELLFLAGS = $@ -ceuconfiguration for a target namedtest, tracked in DOCS-903 with a minimal reproduction.Verified instead with the package tests,
golangci-lint, and the full pre-commit hook.No
--no-verify.Generated with Coder Agents on behalf of @nickvigilante. Linear: DOCS-139