Skip to content

feat: validate documented experiment keys (DOCS-139) - #29316

Merged
nickvigilante merged 7 commits into
mainfrom
vigilante/docs-139-validate-documented-experiment-keys-against-experimentsknown
Oct 1, 2026
Merged

nickvigilante merged 7 commits into
mainfrom
vigilante/docs-139-validate-documented-experiment-keys-against-experimentsknown

Conversation

@nickvigilante

@nickvigilante nickvigilante commented Sep 14, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The docs name specific experiment keys wherever an individual experimental feature is documented, and nothing verifies those keys still exist in the code.
When an experiment is renamed or graduated, the enablement example keeps telling operators to set a key the server no longer accepts.

This adds scripts/checkexperimentkeys, which fails when a key named in docs/** is absent from codersdk.ExperimentsKnown, and wires it into make lint as lint/check-experiment-keys.

The data flows one way.
The check reads keys the docs already name and validates them against code.
It does not generate a page, add a manifest route, or write any key into docs/.
That is deliberate: #29113 retired the generated feature-stage lists because the team decided not to advertise which features are currently experimental or beta.
A guard against stale keys is compatible with that decision; an index is not.

The five key references the check currently guards are the per-feature ones that decision left in place: chat-advisor, chat-virtual-desktop, auto-fill-parameters, agent-lifecycle-hooks, and mcp-server-http.

Recognition rules

Only explicit enablement forms are matched: --experiments=<key-list> and CODER_EXPERIMENTS=<key-list>, including every key in a comma-separated list.
Values may be single- or double-quoted.
Explicit shell line continuations are joined before matching, preserving the first physical line for a finding.
Bare identifiers in prose are ignored on purpose.
A check that guesses at prose produces false positives, and a check with false positives gets deleted by the first person it blocks.
*, feature1, feature2, and angle-bracketed values are recognized placeholders.
The recognized forms are documented in the script's README.

Known gap: the space-separated --experiments <key> form is not matched.
No prose uses it today, so this is latent rather than broken, and closing it is a small follow-up if the form reappears.

Also fixed

ExperimentMCPToolSearch had no case in DisplayName(), so it rendered as a title-cased guess in the product UI as well as anywhere else display names surface.
TestExperimentDisplayNames now pins the display name of every known experiment against an expected map, with a length assertion, so a new experiment cannot silently fall back to the title-cased key.

Verification

  • go test ./scripts/checkexperimentkeys ./codersdk passes.
    Coverage includes known and unknown keys, quoted values, shell continuations, placeholders, multi-key lists, bare-prose exclusion, and failure-output formatting.
  • golangci-lint run ./scripts/checkexperimentkeys/... ./codersdk/... reports 0 issues.
  • make lint/check-experiment-keys is clean against current docs/.
  • Negative control: a fixture containing CODER_EXPERIMENTS=not-a-real-key exits 1 with the expected finding.
  • The full pre-commit hook passes, including lint/check-experiment-keys inside aggregate make lint.
History: the deferred lint wiring, now resolved

This PR previously landed the tool and a standalone make target while leaving the one-line addition to aggregate make lint for a follow-up, and stayed draft, because three documented oauth2 keys were stale and owned by another PR:

  • docs/admin/integrations/oauth2-provider.md:34
  • docs/ai-coder/mcp-server.md:108
  • docs/ai-coder/mcp-server.md:114

#29268 merged on 2026-09-15 and removed all three in favor of CODER_OAUTH2_PROVIDER_ENABLE.
The branch is rebased onto that, the check is green against main, the wiring is folded into this PR, and the merge-order constraint recorded as CRF-5 no longer applies.

The rebase also collided with DisplayName(), which gained new cases on main.
Resolved in favor of main's switch rather than this branch's earlier map refactor, keeping the production diff to the one missing case and moving the exhaustiveness expectations into the test.

Why this replaced a generated reference page

DOCS-139 originally asked for a generated docs/reference/experiments.md listing every key in ExperimentsKnown.
That was built and opened as #29311, then closed without merging.

Commit b23d73d (#29113, six days earlier) had removed both generated feature-stage indexes from docs/install/releases/feature-stages.md, recording that the decision, after discussion with David, was not to advertise either list, and describing them as "drift surfaces that promise a completeness the docs cannot maintain."
It explicitly rejected the superseded #28949, which generated the list from ExperimentsKnown with descriptions parsed from Go constant comments, for making docs track "an internal, unstable surface."

#29311 was that rejected design rebuilt.
The ticket predated the decision; the page should not have been built on top of it.
This PR is the part of the original goal that survives: the drift protection, without the publication.

Local pre-push note

The pre-push hook's test stage cannot run in this workspace: GNU Make 4.4.1 segfaults on the repo's .SHELLFLAGS = $@ -ceu configuration for a target named test, tracked in DOCS-903 with a minimal reproduction.
Verified instead with the package tests, golangci-lint, and the full pre-commit hook.
No --no-verify.


Generated with Coder Agents on behalf of @nickvigilante. Linear: DOCS-139

@linear-code

linear-code Bot commented Sep 14, 2026

Copy link
Copy Markdown

DOCS-139

@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Chat: Review in progress (2/2 reviewers complete) | View chat
Requested: 2026-09-15 00:53 UTC by @nickvigilante

deep-review v0.9.0 | Round 2 | f5be7ae..a2b07d2

Last posted: Round 2, 6 findings (2 P2, 1 P3, 1 Nit, 2 Note), COMMENT. Review

Finding inventory

Finding inventory: PR #29316

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P2 Author fixed (a2b07d2) Makefile:794 PR title scope scripts/checkexperimentkeys omits changed files; title CI check fails R1 Netero Yes
CRF-2 P3 Author fixed (a2b07d2) codersdk/deployment_test.go:34 Loop assertion NotEmpty(DisplayName()) is vacuous; default case guarantees non-empty R1 Netero Yes
CRF-3 Nit Author fixed (a2b07d2) codersdk/deployment_test.go:31 experiment := experiment loop capture unnecessary since Go 1.22 (go.mod: 1.26.5) R1 Netero Yes
CRF-4 Note Author fixed (a2b07d2) scripts/checkexperimentkeys/main.go:106 collectMarkdown 50% / run 75% coverage; walk, dedup, and exit-2 paths untested R1 Netero Yes
CRF-5 Note Author accepted R2 (intentional; resolves when #29268 merges; PR stays draft) Makefile:797 Wiring into make lint turns main lint red if merged before #29268; no merge-order enforcement R1 Netero Yes
CRF-6 P2 Open scripts/checkexperimentkeys/main.go:95 Trailing/adjacent comma yields empty-string key; false-positive unknown experiment key "" fails lint on valid prose R2 Netero Yes

Contested and acknowledged

CRF-5 (Note, Makefile:797) - lint wiring vs #29268 merge order

Round log

Round 1

Netero-only first-pass gate. P2 present (title scope, matches failed title CI check), so panel deferred to next round. 1 P2, 1 P3, 1 Nit, 2 Note. Reviewed against f5be7ae..519ca1e.

Round 2

Churn guard PROCEED: 4 addressed (CRF-1,2,3,4 via a2b07d2), 1 acknowledged (CRF-5). First panel round.
Netero re-run found new P2 (CRF-6, empty-key false positive), verified by reproduction. Pre-panel gate: P2 present, so this is a second Netero-only round; panel deferred to round 3 (consecutive-Netero cap forces panel next round). 1 P2 new. CRF-1..4 fixes verified. CRF-5 still acknowledged. Reviewed against f5be7ae..a2b07d2.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First-pass review only. These are mechanical findings from Netero; the full review panel has not yet reviewed this PR. The panel will review once these are addressed.

The design is well-argued: a one-way drift guard that validates keys the docs already name, deliberately scoped to explicit enablement forms so it cannot produce the false positives that get a check deleted. Test coverage of the core matching logic is real, and the DisplayName fix is backed by an assertion that would fail against the pre-PR default title-casing.

One P2 blocks the mechanical floor: the PR title scope scripts/checkexperimentkeys does not contain every changed file (Makefile, codersdk/), which is why the title CI check is red. Beyond that: one vacuous test assertion, one stale-Go loop capture, and two notes worth a human decision (coverage of the walk/error paths, and the merge-order coupling with #29268 that can turn main's lint red).

As Netero put it: the loop assertion "as written is vacuous; if the intent is to force an explicit case per experiment, it must assert the value differs from the default title-casing."

Severity count: 1 P2, 1 P3, 1 Nit, 2 Note.

🤖 This review was automatically generated with Coder Agents.

Comment thread Makefile Resolved
Comment thread codersdk/deployment_test.go Resolved
Comment thread codersdk/deployment_test.go Resolved
Comment thread scripts/checkexperimentkeys/main.go Resolved
Comment thread Makefile Resolved
@nickvigilante nickvigilante changed the title feat(scripts/checkexperimentkeys): validate documented experiment keys (DOCS-139) feat: validate documented experiment keys (DOCS-139) Sep 15, 2026
@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review, still first-pass only. The full panel has not yet reviewed this PR; a new P2 in the core matcher keeps the mechanical floor unclean, so the panel is deferred one more round and will review once this is addressed.

Round-1 findings resolved and verified: the PR title now omits the scope for a cross-cutting change (CRF-1, title CI green); the vacuous loop assertion is replaced by an internal test that maps display names explicitly and requires one entry per known experiment, rejecting extras (CRF-2); the unnecessary loop capture is gone (CRF-3); and coverage of the walk, dedup, and both exit-2 I/O paths is added, bringing the package to 87% (CRF-4). Good, targeted fixes.

CRF-5 (wiring into make lint turns main red until #29268 merges) remains acknowledged with no ticket. The author's plan is sound (keep the PR draft, target main, wait for #29268), but the merge-order dependency is a human decision: do not merge this before #29268. Not closed.

New this round: one P2. The recognition regex includes , in its character class, so a value followed by ordinary comma punctuation (CODER_EXPERIMENTS=notifications, then restart.) splits into a trailing empty key and reports unknown experiment key "". Reproduced against a one-line Markdown file. As Netero put it, this is "exactly the false-positive failure mode the design states it avoids."

Severity count: 1 P2 (new). CRF-1..4 verified fixed; CRF-5 acknowledged.

🤖 This review was automatically generated with Coder Agents.

Comment thread scripts/checkexperimentkeys/main.go Resolved
@nickvigilante
nickvigilante marked this pull request as ready for review September 22, 2026 17:48
@nickvigilante
nickvigilante force-pushed the vigilante/docs-139-validate-documented-experiment-keys-against-experimentsknown branch from bbeeb1b to 7b57abe Compare September 22, 2026 17:49
@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

…key lint into make lint

The deferred wiring waited on #29268, which merged and removed the three
stale documented oauth2 keys. The check is green against main, so the
standalone target joins aggregate lint.

Keep DisplayName as a switch, matching main, and pin every known
experiment's display name from an expected map in the package test so a
new experiment cannot silently fall back to a title-cased key.

@ianjevans ianjevans left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

The note's CODER_EXPERIMENTS=oauth2 example is an explicit enablement
form, so lint/check-experiment-keys reports the removed experiment as an
unknown key. Describe the removal without naming it as a value.
ExperimentsKnown gained chat-projects, chat-inline-mcp-servers,
enable-ai-workspace-debug, chat-board, and chat-stage-metrics on main,
so TestExperimentDisplayNames needs their expected names.
@nickvigilante
nickvigilante force-pushed the vigilante/docs-139-validate-documented-experiment-keys-against-experimentsknown branch from 7b57abe to 3223e0b Compare October 1, 2026 19:53
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Docs preview

Check off each page once it's been reviewed. If a page changes in a later push, its checkbox clears automatically so it gets a fresh look. Pages not yet wired into the docs navigation aren't listed here.

@nickvigilante
nickvigilante enabled auto-merge (squash) October 1, 2026 20:05
@nickvigilante
nickvigilante merged commit 149366a into main Oct 1, 2026
32 checks passed
@nickvigilante
nickvigilante deleted the vigilante/docs-139-validate-documented-experiment-keys-against-experimentsknown branch October 1, 2026 20:08
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 1, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants