Skip to content

feat: make doc-check review docs quality, not only missing docs - #31193

Merged
nickvigilante merged 2 commits into
mainfrom
vigilante/docs-1410-doc-check-review-docs-for-accuracy-coverage-audience-and
Oct 23, 2026
Merged

nickvigilante merged 2 commits into
mainfrom
vigilante/docs-1410-doc-check-review-docs-for-accuracy-coverage-audience-and

Conversation

@nickvigilante

Copy link
Copy Markdown
Collaborator

Summary

Rebuilds the doc-check skill around one question: if this PR merged now, would the docs on main explain the change correctly and clearly?
On #31159 the current skill stayed silent on real defects, because "a PR that documents itself needs nothing more" ended the review before anything checked the docs the PR added.

Changes

  • doc-check skill: a change model built from the code, eight focused personas with a verifier, findings that must cite evidence or get dropped, explicit blocking classes, a changed-lines rule for style, design-friction notes, and a run trace on every run (output.md, personas/).
  • Rules stay in docs/.style/: each rule page lists its personas in review-personas front matter, so the skill finds rules without depending on file paths.
  • Style guide: new rules for leading with the reader's surface and for time-relative words, a clearer Latin abbreviations rule with Do and Don't examples, and as in the one-meaning table. Coverage counts go from 88 to 90.
  • Conflicting guidance fixed: .claude/docs/DOCS_STYLE_GUIDE.md and the old skill no longer recommend semicolons or "see".
  • write-docs and AGENTS.md: authors run doc-check on their branch before opening or updating a PR.
  • Workflow prompt: asks for the full review, reads the skill from the base branch, and runs persona sub-agents only on the configured model.

Out of scope, tracked separately: commit statuses (DOCS-1407), the replay harness (DOCS-923), draft: true (DOCS-1412), stable rule IDs (DOCS-1413), and "native" wording (DOCS-1409).

Testing

  • make lint/markdown, lint/emdash, lint/style-claims, lint/docs-html, lint/docs-manifest, lint/actions (actionlint and zizmor), and Vale on the changed docs/.style files all pass.
  • Read-only evaluation runs of the new skill, with nothing posted:
Known issue on #31159 Solstice-1 Local, v1 Sonnet 5.5, v1 Solstice-1 Local, v2
Personal and organization downloads lose the manual-only line Filed as pre-existing Blocking Blocking
New section contradicts the Project skills section Missed Missed Blocking
Section leads with the API field, not the UI switch Blocking Blocking Blocking
Quoted "true" is silently ignored Filed as pre-existing Suggestion Missed
Quoted key is stored unchanged, so the file can disagree Blocking Blocking Missed
Claim about other tools can't be checked Suggestion Suggestion Suggestion
PR description claim the code doesn't support Trace Trace Trace
False positives 0 0 0

On #31164, a docs-metadata PR with nothing wrong, v2 reported a clean result with no findings.
v2 fixes the v1 misses: the Continuity Editor reads the whole page first, and gaps on paths the change didn't edit count as caused by the change.
The coverage persona's example comes from #31159, so the v2 rerun on that PR overstates the gain. The replay set in DOCS-923 is the real test.

Risks

  • Eight personas on Solstice may exceed the workflow's 40-minute wait. Watch the first runs after merge.
  • Reviews now post inline comments as well as the summary comment, on every PR that reaches the review.
Plan and decision log

doc-check redesign: decisions so far

Status: design interview in progress. Nothing implemented.
Tracking: DOCS-918 (parent), DOCS-1407 (commit status), DOCS-925 (outcomes), DOCS-923 (replay), DOCS-926 (harvester), DOCS-1378 (blocking slop), DOCS-597 (renderer class drift).

First principle

If this PR merged now, would the docs on main correctly and clearly explain this exact change?
The requirement runs both ways: every docs claim is grounded in the code, and every behavior that materially changes how a customer uses the product has docs.
"The PR documents itself" is never a reason to stop reviewing.

Scope and audience

  • One skill answers both "are docs needed" and "are the added docs good".
  • Every PR gets the same depth, including docs-only PRs.
  • Findings are written for the PR author, and phrased so an authoring agent can apply them.

Priority and blocking

Order: accuracy, journey completeness, consistency, audience fit, structure, concision, word choice and mechanics.

Blocking findings:

  • Claims the code contradicts.
  • Required docs that are missing for behavior the PR adds or changes.
  • Rendering breaks and typos (deterministic checks first, LLM as backup).
  • Consistency and audience findings only when all three hold: the finding cites a rule ID or a contradicting line, the verifier confirmed it, and the PR added or edited the text.

Never blocking:

  • Pure judgment calls without a citable rule. To make something blockable, add a rule.
  • Style issues on lines the PR did not touch.
  • Design-friction notes (uncapped, own category, each tied to a docs sentence, re-evaluated every round).
  • Gaps that predate the PR: filed to Linear through the DOCS-926 harvester.
  • Tutorial or lab needs: filed to TRAIN or DOCS through the harvester.

Evidence and hallucination handling

  • Every finding cites evidence: code location, other page and line, or style rule ID. No evidence, no finding.
  • Verifier persona re-checks every finding on every run.
  • The summary lists checks the reviewer could not complete.
  • /doc-check wrong <finding-id> <reason> logs a false finding to Linear and adds the PR to the replay set.

Output

  • One PR review per round with as many inline comments as needed, suggestion blocks for local fixes.
  • Re-reviews comment only on changed lines and resolve fixed threads. A clean round posts nothing new.
  • Hidden machine-readable findings block (ID, file, line, rule ID, blocking, fix) for agent fix loops.
  • Every run leaves a trace in the commit status and job summary, even with zero findings.

Commit status and override (DOCS-1407)

  • docs status on every commit, outcomes finding, clean, error, skipped.
  • An approving review from a @coder/docs member overrides to success.
  • The override carries forward while later commits pass.
  • A later failing commit re-requests review from the approver, then @coder/docs after one business day. The engineer can still merge.
  • Rollout: report-only for a few weeks, then required for accuracy and missing docs once blocking false positives are at or under 5%, then for rule-anchored consistency and audience.

Personas

One orchestrator chat with sub-agents, each loading only its rule slice.
IDs match the job; display names are editorial titles.

ID Name Job
accuracy Fact Checker Every docs claim is supported by the code
coverage Cartographer Undocumented behavior, and docs for nonexistent behavior
learner Newcomer Reads only post-merge docs, reports what it learned, compared against a code-derived change summary
consistency Continuity Editor Terms and facts agree across and within pages
audience Reader Advocate UI before API, reader task before internals, page type fits
structure Architect Headings, procedures, prerequisites, placement
language Copy Editor Word choice, sentence shape, mechanics beyond Vale
verifier Skeptic Re-checks every finding's evidence

Language and consistency can skip when the PR changes no prose.
The persona list grows as needed.

Rules

  • docs/.style/ is the single human-readable rule source. Fix conflicting text in .claude/docs/DOCS_STYLE_GUIDE.md and the doc-check skill first (semicolons, unit spacing, "see").
  • Each style-guide page lists review-personas in front matter. Each rule heading carries a stable ID anchor. The bot resolves rules by ID and front matter, never by path.
  • Calibration answers that differ from the guide become style-guide PRs before a persona relies on them, and the calibration sentences become persona test cases.

Ethos captured so far

  • Test for implementation detail: keep it only if it changes what the reader sees, types, or must do.
  • UI label leads; the API field appears once as "In the API, ..."; frontmatter keys only where the reader types them.
  • "Did I learn the thing this PR changes?" is the core reading test.
  • Always block: typos, markup that breaks rendering, wrong claims, unsupported compliance claims, multiple actions in one step (STE), "See [link]" with non-descriptive link text, missing prerequisites, inconsistent terms, user-facing implementation narrative, missing docs for a flag that changes a user task.
  • Suggest: "simply" (rewrite the step), passive voice (make the actor the subject), long multi-condition sentences, semicolons, "now" (replace with the condition that makes the action possible), as or since for cause (use "because"), niche flags without guide coverage.
  • Latin: never block. On a line that changes for another reason, suggest "for example" in place of "e.g.". The accepted forms are (e.g., ...) and (for example, ...).

Architecture

  • coder/coder holds the Markdown: skill, personas, rules.
  • Private coder/doc-check holds everything else as a hosted GitHub App: orchestration, status, override, re-request, slash commands, verdict parsing, replay harness, persona-to-model map.
  • Authoring agents run the same skill locally before opening or updating a PR (required in AGENTS.md and write-docs).
  • Review chats are shared with @coder/docs, not the whole organization.

Models

  • Live PRs: Solstice-1 with cloud fallback. Record whether fallback happened on each run and why.
  • Replay and calibration: Solstice-1 Local (no cloud fallback) so data stays clean.
  • Persona-to-model map in the bot repo. A persona moves to a different local model only when replay scores show it is better for that persona.
  • Opus only through /doc-check model:opus and in the manual replay.

Replay set (DOCS-923)

Calibration set 2 decisions

  • Open PRs that would change a claim in this PR are scanned and named in the summary. They never block.
  • External claims (other products) never block. Suggest narrowing to what was checked, linking vendor docs, or adding a product test that fails if the claim breaks.
  • New rule: when a docs claim depends on product behavior, recommend a test in the product that fails if the claim stops being true.
  • New page rule: block a page that is not in docs/manifest/ unless its front matter has draft: true, and block a page that is in the manifest while its front matter has draft: true. No page on main uses draft: true and the manifest tooling doesn't read it today, so the convention needs defining first.
  • Admin-only settings: block when they're documented only under docs/user-guides/, and require that some page covers them.
  • Stale screenshots: block when the image contradicts a step on the page, otherwise harvest. Long term, generated screenshots replace hand captures (P-DOCS-174, DOCS-1408).
  • Inclusive wording: replace "non-native readers" in the style guide with wording about readability and translation.

Open

  • Calibration set 2: items Nick didn't answer keep my bracketed guess.
  • How coder-agents-review-bot is hosted, to model the doc-check service on it.
  • How to detect cloud fallback per chat (per-message model config or provider metadata).
  • Commit status versus check run.

Overnight decisions (made without Nick; override any of these)

Each entry: the decision, why, and how to undo it.

Linear

  1. DOCS-1409 "native" wording: separate issue in the Docs style guide project, Medium, 2 points. Not fixed in the PR, because Nick asked for a separate issue. The Latin rewrite drops one "non-native readers" sentence only because that paragraph was rewritten anyway.
  2. DOCS-1412 draft: true: sub-issue of DOCS-918, Medium, 3 points. Scoped as a deterministic check in docsmanifestgen, not an LLM rule, because blocking rules should be deterministic.
  3. DOCS-1411 Pixel screenshots: in Automated screenshot capture (P-DOCS-174), Medium, 5 points, related to DOCS-45 and DOCS-1408. Pixel replaces the Playwright proof of concept for anything a Storybook story renders; Playwright stays for flows that need a live deployment. The project summary is unchanged.
  4. DOCS-1410 for this PR: sub-issue of DOCS-918, High, 5 points, In Progress.
  5. DOCS-1413 stable rule IDs: deferred out of this PR, because IDs for 90 rules plus a styleclaims check is its own reviewable change.

PR scope

  1. One PR for skills plus style guide, because the persona front matter and the skill depend on each other. Out of scope: the hosted bot, commit statuses, replay harness, slash commands, sharing chats with @coder/docs.
  2. Personas are files in .claude/skills/doc-check/personas/ with the agreed names.
  3. review-personas front matter on content-guidelines.md and eight style-guide pages, documented in docs/.style/README.md. Single-line list form so a plain grep finds it. editor-setup.md and the READMEs get none, because they hold no rules.
  4. Mapping: content guidelines to accuracy, coverage, audience, structure. Audience and scope to audience, learner, structure. Word choice to language, consistency, accuracy. Procedural writing and formatting to structure, language. Capitalization and punctuation to language, structure. Accessibility and inclusion to language, structure, audience. Voice and tone, and numbers, to language.
  5. CI reads the skill from the base branch, so a PR can't rewrite its own review. Author self-check reads it from the branch.
  6. Sub-agents run only when the workflow passes a model config ID (vars.DOC_CHECK_MODEL_CONFIG_ID), so a run never mixes models. Otherwise personas run sequentially.
  7. Posting: one COMMENT review with inline comments, plus the existing sticky summary comment and marker, because the Slack notice and re-review logic depend on the marker. Findings for agents go in a collapsed YAML block in the summary.
  8. The footer tells authors to reply in the thread when a finding is wrong. /doc-check wrong isn't advertised, because it doesn't exist yet.
  9. The run trace is a YAML block at the end of the final chat message, so DOCS-1407 can parse it into a commit status later.
  10. Deterministic checks run first. A failing check on a changed line blocks.
  11. The 40-minute wait timeout is unchanged. Eight personas on Solstice may run longer, so watch for timeouts after merge. Raising it is a one-line change.

Style guide content

  1. Latin abbreviations: (e.g., ...) and (for example, ...) are both accepted. The comma form is a Don't example but never blocks. Reviewers suggest the English form only when the line changes for another reason. etc. stays accepted in prose, because Ben's position was to accept Latin abbreviations. Nick's personal "tables only" preference is not encoded.
  2. as row added to Choose words that have one meaning, documentation-only, because as has too many correct uses for a Vale word list.
  3. New rule: Describe conditions, not moments in time (voice and tone). now stays allowed for progress inside a sequenced tutorial, because word choice already uses "Now that you added a language".
  4. New rule: Lead with the surface the reader uses (audience and scope), with the feat: honor disable-model-invocation in uploaded skill content #31159 sentence as the Don't example. Tutorials that mirror UI and CLI paths keep both.
  5. Coverage counts updated from 88 to 90 rules. make lint/style-claims verifies them.
  6. Content guidelines: one paragraph under Programmatic content asking for a product test when no test pins a documented behavior, which backs the "recommend a test" rule.
  7. .claude/docs/DOCS_STYLE_GUIDE.md: removed the semicolon recommendation, "See" as a navigation verb (kept "See which applications", which describes the UI), and a bare 150ms threshold (now in code font, matching the content guidelines).
  8. write-docs: whole-path tracing, checking the PR description against the code, naming the persona, leading with the reader's surface, running doc-check before the PR, two anti-patterns, two checklist items, and a fix for the stale "Premium pages carry the title suffix" item.
  9. AGENTS.md: one workflow bullet requiring the doc-check self-check before PRs that touch docs/ or user-facing behavior.

Evaluation

  1. Eval set: feat: honor disable-model-invocation in uploaded skill content #31159 (known defects) and docs: remove beta label from JetBrains Toolbox #31164 (known clean). Solstice-1 Local for tuned runs, Sonnet 5.5 as the comparison. PR heads checked out read-only, nothing posted.
  2. Changes after the first eval: both models missed the contradiction between the new section and an older section of the same page, and Solstice filed the download gap and the silently ignored quoted value as adjacent. The Continuity Editor now reads the whole page first, "the change caused it" includes paths the change didn't edit, and only class 4 findings need a style rule to block. Risk: the coverage example mentions downloads, which comes from feat: honor disable-model-invocation in uploaded skill content #31159, so the v2 rerun on feat: honor disable-model-invocation in uploaded skill content #31159 overstates the improvement. The replay set (DOCS-923) is the real test.

Generated by Coder Agents on behalf of @nickvigilante.

@linear-code

linear-code Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

DOCS-1408

DOCS-45

DOCS-1410

@nickvigilante
nickvigilante marked this pull request as ready for review October 23, 2026 14:51
@nickvigilante
nickvigilante requested review from bpmct and removed request for jdomeracki-coder October 23, 2026 14:51
@coder
coder Bot requested a review from ianjevans October 23, 2026 15:48
doc-check stayed silent on #31159 because "a PR that documents itself
needs nothing more" ended the review before anything checked the docs
the PR added, and it reported a check it never ran. Of the last 60
non-docs-team PRs it commented on 2.

The skill now answers one question: if this change merged now, would
the docs on main explain it correctly and clearly? It runs a code-only
change model, eight focused personas with a verifier, evidence-or-drop
findings, explicit blocking classes, a changed-lines rule for style,
and a run trace on every run. Personas find their rules through
review-personas front matter in docs/.style, so the style guide can be
reorganized without breaking the review.

write-docs and AGENTS.md now ask authors to run doc-check on their
branch first. The style guide gains rules for leading with the
reader's surface and for time-relative words, clarifies Latin
abbreviations, adds "as" to the one-meaning table, and
DOCS_STYLE_GUIDE.md stops recommending semicolons and "see".

Refs DOCS-1410.
…s on unedited paths

A first evaluation on #31159 with Solstice-1 Local and Sonnet 5.5
showed both models missing a new section that contradicted an older
section of the same page, and Solstice filing a download gap the PR
caused as a pre-existing gap. The Continuity Editor now reads the
whole page first, coverage and the verifier count gaps on paths the
change did not edit, and only rule-anchored findings need a style rule
to block.

Refs DOCS-1410.
@nickvigilante
nickvigilante force-pushed the vigilante/docs-1410-doc-check-review-docs-for-accuracy-coverage-audience-and branch from 6ecaaf9 to 1ba86a7 Compare October 23, 2026 16:52
@nickvigilante
nickvigilante enabled auto-merge (squash) October 23, 2026 16:52
@nickvigilante
nickvigilante merged commit c242029 into main Oct 23, 2026
30 of 31 checks passed
@nickvigilante
nickvigilante deleted the vigilante/docs-1410-doc-check-review-docs-for-accuracy-coverage-audience-and branch October 23, 2026 17:16
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 23, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants