Skip to content

ci: report only wrong, missing, and broken findings in CI doc-check reviews - #32199

Merged
nickvigilante merged 1 commit into
mainfrom
vigilante/docs-1752-doc-check-report-only-wrong-missing-and
Nov 13, 2026
Merged

nickvigilante merged 1 commit into
mainfrom
vigilante/docs-1752-doc-check-report-only-wrong-missing-and

Conversation

@nickvigilante

@nickvigilante nickvigilante commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

A CI doc-check review now runs three lenses and the verifier, and posts only wrong, missing, and broken findings.
The four style lenses stop running on each push.

This applies the scope decision of 2026-11-13.
The plan gated it on a finding-level eval, which does not exist yet, so the effect on what the review catches is not measured.

Changes

In CI review mode only:

  • Run the Fact Checker, the Cartographer, the Newcomer, and the Skeptic.
  • Do not run the Continuity Editor, the Reader Advocate, the Architect, or the Copy Editor.
  • Keep only wrong, missing, and broken findings.
    A rule, suggestion, or friction finding is dropped before verification, so the verifier also gets fewer groups.
  • Blocking class 4, rule-anchored, does not apply.

The four lenses that leave also found a few things that are wrong or broken, not style.
Those checks move to a section named Also in CI review in two persona files:

Check From To
Two statements that give different values, options, or steps for one thing Continuity Editor Fact Checker
A UI label that does not match the component Continuity Editor Fact Checker
Typos Copy Editor Fact Checker
Links and anchors the change adds Architect Cartographer
Markup the renderer does not support Architect Cartographer
A missing state: on a staged feature Architect Cartographer

The Cartographer already checks manifest entries, redirects, and a role or license missing before the step that needs it.

Also changed:

  • The workflow prompt said "every persona". It now says every persona that the CI review mode runs.
  • docs/.style/README.md says which personas a CI review runs, because a rule page's review-personas field decides whether CI applies it.

The author self-check and the focused review do not change: they run every lens and apply every class.

Why

Reviews with 8 or more sub-agents were 8 percent of reviews and 38 percent of tokens on the shared model.
A review of a docs change could start up to seven lens sub-agents, and now starts up to three.
Each model call that a review no longer makes is also one less call that can wait in the queue.

Testing

Static: markdownlint-cli2, markdown-table-formatter --check, actionlint, make lint/emdash, and go run ./scripts/styleclaims pass.

Dry run: two replays of a CI review of #31159 at 3580c80, with this skill text plus the step 3 change from #32197, posting nothing.
Limits of this test: one PR, which the skill's own examples come from; a cloud model, not the shared model; one agent running every persona in turn, with no sub-agents.

  • Both replays ran only the four personas, skipped the deterministic checks, ran no lint or test, and dropped the suggestion-class findings.
  • The first replay did not run the moved checks, which were then a list in SKILL.md.
    I moved them into the persona files, and the second replay ran five of the six and said what it compared.
    It judged the feature state check not applicable without opening the manifest.
  • Three findings on this head were known from earlier reviews:
Known finding Result
Download loses the manual-only line Found, as missing
The new section and line 257 list different ways to invoke the skill Not reported. Both sentences are true in the code, so it is not wrong.
The API field leads the UI switch Not looked for. It is a style finding, out of scope by design.
  • Both replays also reported "the stored file never disagrees with the setting" as wrong, with code cited.
    I did not check that finding against the code myself.

Not tested:

  • A live CI review.
    A CI review reads the skill from the base branch, so this text takes effect only after it merges.
  • Any review on the shared model, and any review that uses sub-agents.

What this stops catching in CI

  • Every style-guide and content-guideline rule that the style lenses applied by reading.
  • A same-page inconsistency where neither sentence is false, as in the second row above.
  • Suggestions, including a test for a fragile claim, and design-friction notes.

Linear: DOCS-1752

Decision log
  • The Newcomer stays.
    It returns only missing and wrong, so it is in scope.
    The design moves it to large docs changes only, after an eval says whether it finds what the facts check does not.
  • The moved checks live in the persona files, not in SKILL.md.
    A reviewer follows the method in the persona file it is running.
    The first replay showed that a list elsewhere was read and not applied.
  • Suggestions are dropped, not demoted.
    The decision was that per-push CI reports three classes.
    Dropping them before verification also removes verifier work.
  • No tiers and no sub-agent cap in this PR.
    Those change how the review is structured, and stay behind the eval.
  • Threads and summary items of other classes from earlier reviews are left alone.
    A CI review can no longer tell whether a style finding was fixed.

Generated by Coder Agents on behalf of @nickvigilante.

…eviews

A CI review of a docs change ran seven lenses and one verifier per group
of findings. Reviews with 8 or more sub-agents were 8 percent of reviews
and 38 percent of tokens on the shared model.

A CI review now runs the Fact Checker, the Cartographer, the Newcomer,
and the Skeptic, and posts only wrong, missing, and broken findings. The
wrong and broken checks that the four other lenses carried move to the
Fact Checker and the Cartographer. The author self-check still runs
every lens and applies every class.

Linear: DOCS-1752
@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 21:56 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 21:56 — with GitHub Actions Inactive
@linear-code

linear-code Bot commented Nov 13, 2026

Copy link
Copy Markdown

DOCS-1752

@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 21:56 — with GitHub Actions Inactive
nickvigilante added a commit that referenced this pull request Nov 13, 2026
…s to CI in CI reviews (#32197)

## Summary

Step 3 of the doc-check skill has the review run lint that CI already
runs on the same PR.
In CI review mode the skill now skips that step, and runs no test suite,
build, or generator.

## Changes

- `SKILL.md` step 3 starts with a CI review rule: skip the step, run no
test suite, build, or generator, and record the skip in the trace.
- The other modes keep the step, because an author runs them before CI
does.

Every check the step lists already runs in `ci.yaml`:

| Check | CI job |
|-------|--------|
| `make lint/emdash`, `make lint/docs-html`, `make lint/docs-manifest`,
`make lint/markdown` | `lint`, through `make lint` |
| markdownlint and the table format check | `lint-docs`, through `pnpm
check-docs` |
| Vale, advisory | `lint-docs` |

Both jobs are in the `required` check, so a failing check blocks the
merge without the review reporting it.

## Why

On 2026-11-13 a review of #32185 took 215 seconds.
113 of them went to two tool calls that ran lint and the PR's own two
shell test suites, each suite twice.
Each of those calls is also one more model call that can wait in the
queue.

## Testing

- `markdownlint-cli2` and `markdown-table-formatter --check` pass on
`SKILL.md`.
- Not tested in a live review.
A CI review reads the skill from the base branch, so this text takes
effect only after it merges.
- Dry run: two replays of a CI review of #31159 at `3580c80` with this
text plus #32199, posting nothing.
Both skipped step 3, ran no lint, test, or build, and wrote the
`deterministic_checks` entry as the step asks.
The replays ran on a cloud model with no sub-agents, not on the shared
model.

## Not in this PR

- Passing CI's check results into the review prompt.
- The Copy Editor persona asks for these results so it does not repeat
them, and a CI review now has none to give it.

Linear: DOCS-1751

_Generated by Coder Agents on behalf of @nickvigilante._
@nickvigilante
nickvigilante marked this pull request as ready for review November 13, 2026 22:15
@nickvigilante
nickvigilante requested a review from bpmct November 13, 2026 22:15
@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 22:15 — with GitHub Actions Inactive

@bpmct bpmct left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, approving. Thanks for the doc-check improvements.

Generated by Coder Agents on behalf of @bpmct.

@nickvigilante
nickvigilante merged commit a921700 into main Nov 13, 2026
64 of 65 checks passed
@nickvigilante
nickvigilante deleted the vigilante/docs-1752-doc-check-report-only-wrong-missing-and branch November 13, 2026 22:31
@nickvigilante
nickvigilante temporarily deployed to docs-preview November 13, 2026 22:31 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 22:31 — with GitHub Actions Inactive
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 13, 2026

This branch was previously deployed

2 inactive deployments
docs-preview — e39cab91 Deployed Nov 13, 2026 by nickvigilante via cleanup #64
main — e39cab91 Deployed Nov 13, 2026 by nickvigilante via cla #257043
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants