Repository navigation
feat: validate documented experiment keys (DOCS-139) #29316
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
nickvigilante
merged 7 commits into
main
from
vigilante/docs-139-validate-documented-experiment-keys-against-experimentsknown
Oct 1, 2026
Merged
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
a29e0bd
feat: validate documented experiment keys (DOCS-139)
nickvigilante 39f06e4
fix: cover experiment key validation edge cases (DOCS-139)
nickvigilante e93b5a2
fix(scripts/checkexperimentkeys): ignore empty experiment keys
nickvigilante cad0d28
fix: defer experiment key lint wiring
nickvigilante 5493532
feat(Makefile,codersdk,scripts/checkexperimentkeys): wire experiment …
nickvigilante d0d6b1d
docs(docs/admin/integrations/oauth2-provider): drop the oauth2 example
nickvigilante 3223e0b
test(codersdk): pin display names for the newer experiments
nickvigilante File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Some comments aren't visible on the classic Files Changed page.
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| # checkexperimentkeys | ||
|
|
||
| `checkexperimentkeys` validates experiment keys explicitly named in Markdown | ||
| under `docs/` against `codersdk.ExperimentsKnown`. It runs as | ||
| `make lint/check-experiment-keys`, which is part of `make lint`. | ||
|
|
||
| ## Recognized forms | ||
|
|
||
| The check only recognizes explicit enablement examples: | ||
|
|
||
| - `--experiments=<key>` | ||
| - `CODER_EXPERIMENTS=<key>` | ||
| - comma-separated key lists in either form, such as | ||
| `--experiments=key-one,key-two` | ||
| - single- or double-quoted values, such as `CODER_EXPERIMENTS="key-one"` | ||
| - shell line continuations that start with one of the forms above | ||
|
|
||
| It deliberately does not try to infer bare identifiers from prose. The | ||
| placeholder values `*`, the documented illustrative pair `feature1` and | ||
| `feature2`, and any angle-bracketed value such as `<experiment-key>` or | ||
| `<key>` are ignored. Other `featureN` values are checked so a real stale key | ||
| is not treated as a placeholder. | ||
|
|
||
| Directory symlinks are not traversed. This keeps a directory scan contained to | ||
| the explicitly supplied tree and prevents a documentation checkout from | ||
| silently scanning Markdown outside it. A Markdown file supplied directly is | ||
| still scanned according to normal file access rules. | ||
|
|
||
| ## Usage | ||
|
|
||
| ```console | ||
| $ go run ./scripts/checkexperimentkeys # scans docs/ | ||
| $ go run ./scripts/checkexperimentkeys path/to/file.md path/to/dir | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,175 @@ | ||
| // Command checkexperimentkeys validates experiment keys named in documentation. | ||
| // | ||
| // The check recognizes only explicit enablement forms: --experiments=<key-list> | ||
| // and CODER_EXPERIMENTS=<key-list>, where a key list is comma-separated. It | ||
| // deliberately does not infer bare identifiers from prose, which would make | ||
| // the check susceptible to false positives. Matching single- or double-quoted | ||
| // values are accepted. Placeholders are *, the documented feature1 and | ||
| // feature2 illustrative pair, and any angle-bracketed value. | ||
| // | ||
| // Usage: | ||
| // | ||
| // checkexperimentkeys [path ...] | ||
| // | ||
| // With no arguments it scans docs/. Arguments may be files or directories. | ||
| package main | ||
|
|
||
| import ( | ||
| "fmt" | ||
| "io" | ||
| "io/fs" | ||
| "os" | ||
| "path/filepath" | ||
| "regexp" | ||
| "slices" | ||
| "sort" | ||
| "strings" | ||
|
|
||
| "github.com/coder/coder/v2/codersdk" | ||
| ) | ||
|
|
||
| var ( | ||
| experimentAssignment = regexp.MustCompile(`(?:--experiments|CODER_EXPERIMENTS)=(?:"([a-zA-Z0-9_*<>,-]+)"|'([a-zA-Z0-9_*<>,-]+)'|([a-zA-Z0-9_*<>,-]+))`) | ||
| shellContinuationAssignment = regexp.MustCompile(`(?:--experiments|CODER_EXPERIMENTS)=[a-zA-Z0-9_*<>,-]*\\$`) | ||
| ) | ||
|
|
||
| type finding struct { | ||
| path string | ||
| line int | ||
| key string | ||
| } | ||
|
|
||
| func main() { | ||
| roots := os.Args[1:] | ||
| if len(roots) == 0 { | ||
| roots = []string{"docs"} | ||
| } | ||
| os.Exit(run(roots, os.Stderr)) | ||
| } | ||
|
|
||
| // run scans roots, writes findings to stderr, and returns 0 for success, 1 for | ||
| // unknown keys, or 2 for I/O errors. | ||
| func run(roots []string, stderr io.Writer) int { | ||
| return runWithReadFile(roots, stderr, os.ReadFile) | ||
| } | ||
|
|
||
| func runWithReadFile(roots []string, stderr io.Writer, readFile func(string) ([]byte, error)) int { | ||
| files, err := collectMarkdown(roots) | ||
| if err != nil { | ||
| _, _ = fmt.Fprintf(stderr, "checkexperimentkeys: %v\n", err) | ||
| return 2 | ||
| } | ||
|
|
||
| known := make(map[string]bool, len(codersdk.ExperimentsKnown)) | ||
| for _, experiment := range codersdk.ExperimentsKnown { | ||
| known[string(experiment)] = true | ||
| } | ||
|
|
||
| var findings []finding | ||
| for _, path := range files { | ||
| src, err := readFile(path) | ||
| if err != nil { | ||
| _, _ = fmt.Fprintf(stderr, "checkexperimentkeys: %v\n", err) | ||
| return 2 | ||
| } | ||
| findings = append(findings, checkSource(path, src, known)...) | ||
| } | ||
|
|
||
| for _, finding := range findings { | ||
| _, _ = fmt.Fprintf(stderr, "%s:%d: unknown experiment key %q; add it to codersdk.ExperimentsKnown or remove or update this documented enablement example.\n", finding.path, finding.line, finding.key) | ||
| } | ||
| if len(findings) > 0 { | ||
| _, _ = fmt.Fprintf(stderr, "\ncheckexperimentkeys: found %d documented experiment key(s) that are not in codersdk.ExperimentsKnown.\n", len(findings)) | ||
| return 1 | ||
| } | ||
|
|
||
| return 0 | ||
| } | ||
|
|
||
| func checkSource(path string, src []byte, known map[string]bool) []finding { | ||
| var findings []finding | ||
| for _, line := range shellLogicalLines(string(src)) { | ||
| for _, match := range experimentAssignment.FindAllStringSubmatch(line.text, -1) { | ||
| for _, key := range strings.Split(experimentValue(match), ",") { | ||
| if key == "" || known[key] || isPlaceholder(key) { | ||
| continue | ||
| } | ||
| findings = append(findings, finding{path: path, line: line.number, key: key}) | ||
| } | ||
| } | ||
| } | ||
| return findings | ||
| } | ||
|
|
||
| type logicalLine struct { | ||
| number int | ||
| text string | ||
| } | ||
|
|
||
| func shellLogicalLines(src string) []logicalLine { | ||
| physicalLines := strings.Split(src, "\n") | ||
| logicalLines := make([]logicalLine, 0, len(physicalLines)) | ||
| for lineIndex := 0; lineIndex < len(physicalLines); lineIndex++ { | ||
| line := physicalLines[lineIndex] | ||
| logicalLine := logicalLine{number: lineIndex + 1, text: line} | ||
|
|
||
| // Markdown also uses a trailing backslash as a hard line break. Only join | ||
| // a line ending in an explicit, unquoted assignment the recognizer accepts. | ||
| if shellContinuationAssignment.MatchString(logicalLine.text) { | ||
| for lineIndex+1 < len(physicalLines) { | ||
| lineIndex++ | ||
| logicalLine.text = strings.TrimSuffix(logicalLine.text, "\\") + physicalLines[lineIndex] | ||
| if !strings.HasSuffix(logicalLine.text, "\\") { | ||
| break | ||
| } | ||
| } | ||
| } | ||
| logicalLines = append(logicalLines, logicalLine) | ||
| } | ||
| return logicalLines | ||
| } | ||
|
|
||
| func experimentValue(match []string) string { | ||
| for _, value := range match[1:] { | ||
| if value != "" { | ||
| return value | ||
| } | ||
| } | ||
| return "" | ||
| } | ||
|
|
||
| func isPlaceholder(key string) bool { | ||
| return key == "*" || key == "feature1" || key == "feature2" || | ||
| (strings.HasPrefix(key, "<") && strings.HasSuffix(key, ">")) | ||
| } | ||
|
|
||
| func collectMarkdown(roots []string) ([]string, error) { | ||
|
nickvigilante marked this conversation as resolved.
Resolved
|
||
| var files []string | ||
| for _, root := range roots { | ||
| info, err := os.Stat(root) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| if !info.IsDir() { | ||
| if filepath.Ext(root) == ".md" { | ||
| files = append(files, root) | ||
| } | ||
| continue | ||
| } | ||
| err = filepath.WalkDir(root, func(path string, entry fs.DirEntry, err error) error { | ||
| if err != nil { | ||
| return err | ||
| } | ||
| if entry.IsDir() || filepath.Ext(path) != ".md" { | ||
| return nil | ||
| } | ||
| files = append(files, path) | ||
| return nil | ||
| }) | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
| } | ||
| sort.Strings(files) | ||
| return slices.Compact(files), nil | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.