Repository navigation
ci: report only wrong, missing, and broken findings in CI doc-check reviews - #32199
Merged
nickvigilante merged 1 commit intoNov 13, 2026
Merged
nickvigilante merged 1 commit into
nickvigilante merged 1 commit into
Conversation
…eviews A CI review of a docs change ran seven lenses and one verifier per group of findings. Reviews with 8 or more sub-agents were 8 percent of reviews and 38 percent of tokens on the shared model. A CI review now runs the Fact Checker, the Cartographer, the Newcomer, and the Skeptic, and posts only wrong, missing, and broken findings. The wrong and broken checks that the four other lenses carried move to the Fact Checker and the Cartographer. The author self-check still runs every lens and applies every class. Linear: DOCS-1752
nickvigilante
added a commit
that referenced
this pull request
Nov 13, 2026
…s to CI in CI reviews (#32197) ## Summary Step 3 of the doc-check skill has the review run lint that CI already runs on the same PR. In CI review mode the skill now skips that step, and runs no test suite, build, or generator. ## Changes - `SKILL.md` step 3 starts with a CI review rule: skip the step, run no test suite, build, or generator, and record the skip in the trace. - The other modes keep the step, because an author runs them before CI does. Every check the step lists already runs in `ci.yaml`: | Check | CI job | |-------|--------| | `make lint/emdash`, `make lint/docs-html`, `make lint/docs-manifest`, `make lint/markdown` | `lint`, through `make lint` | | markdownlint and the table format check | `lint-docs`, through `pnpm check-docs` | | Vale, advisory | `lint-docs` | Both jobs are in the `required` check, so a failing check blocks the merge without the review reporting it. ## Why On 2026-11-13 a review of #32185 took 215 seconds. 113 of them went to two tool calls that ran lint and the PR's own two shell test suites, each suite twice. Each of those calls is also one more model call that can wait in the queue. ## Testing - `markdownlint-cli2` and `markdown-table-formatter --check` pass on `SKILL.md`. - Not tested in a live review. A CI review reads the skill from the base branch, so this text takes effect only after it merges. - Dry run: two replays of a CI review of #31159 at `3580c80` with this text plus #32199, posting nothing. Both skipped step 3, ran no lint, test, or build, and wrote the `deterministic_checks` entry as the step asks. The replays ran on a cloud model with no sub-agents, not on the shared model. ## Not in this PR - Passing CI's check results into the review prompt. - The Copy Editor persona asks for these results so it does not repeat them, and a CI review now has none to give it. Linear: DOCS-1751 _Generated by Coder Agents on behalf of @nickvigilante._
nickvigilante
marked this pull request as ready for review
November 13, 2026 22:15
nickvigilante
deleted the
vigilante/docs-1752-doc-check-report-only-wrong-missing-and
branch
November 13, 2026 22:31
nickvigilante
temporarily deployed
to
docs-preview
November 13, 2026 22:31 — with
GitHub Actions
Inactive
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A CI doc-check review now runs three lenses and the verifier, and posts only
wrong,missing, andbrokenfindings.The four style lenses stop running on each push.
This applies the scope decision of 2026-11-13.
The plan gated it on a finding-level eval, which does not exist yet, so the effect on what the review catches is not measured.
Changes
In CI review mode only:
wrong,missing, andbrokenfindings.A
rule,suggestion, orfrictionfinding is dropped before verification, so the verifier also gets fewer groups.The four lenses that leave also found a few things that are wrong or broken, not style.
Those checks move to a section named Also in CI review in two persona files:
state:on a staged featureThe Cartographer already checks manifest entries, redirects, and a role or license missing before the step that needs it.
Also changed:
docs/.style/README.mdsays which personas a CI review runs, because a rule page'sreview-personasfield decides whether CI applies it.The author self-check and the focused review do not change: they run every lens and apply every class.
Why
Reviews with 8 or more sub-agents were 8 percent of reviews and 38 percent of tokens on the shared model.
A review of a docs change could start up to seven lens sub-agents, and now starts up to three.
Each model call that a review no longer makes is also one less call that can wait in the queue.
Testing
Static:
markdownlint-cli2,markdown-table-formatter --check,actionlint,make lint/emdash, andgo run ./scripts/styleclaimspass.Dry run: two replays of a CI review of #31159 at
3580c80, with this skill text plus the step 3 change from #32197, posting nothing.Limits of this test: one PR, which the skill's own examples come from; a cloud model, not the shared model; one agent running every persona in turn, with no sub-agents.
SKILL.md.I moved them into the persona files, and the second replay ran five of the six and said what it compared.
It judged the feature state check not applicable without opening the manifest.
missingwrong.wrong, with code cited.I did not check that finding against the code myself.
Not tested:
A CI review reads the skill from the base branch, so this text takes effect only after it merges.
What this stops catching in CI
Linear: DOCS-1752
Decision log
It returns only
missingandwrong, so it is in scope.The design moves it to large docs changes only, after an eval says whether it finds what the facts check does not.
SKILL.md.A reviewer follows the method in the persona file it is running.
The first replay showed that a list elsewhere was read and not applied.
The decision was that per-push CI reports three classes.
Dropping them before verification also removes verifier work.
Those change how the review is structured, and stay behind the eval.
A CI review can no longer tell whether a style finding was fixed.
Generated by Coder Agents on behalf of @nickvigilante.