Repository navigation
feat: make doc-check review docs quality, not only missing docs - #31193
Merged
nickvigilante merged 2 commits intoOct 23, 2026
Merged
nickvigilante merged 2 commits into
nickvigilante merged 2 commits into
Conversation
nickvigilante
marked this pull request as ready for review
October 23, 2026 14:51
nickvigilante
requested review from
bpmct
and removed request for
jdomeracki-coder
October 23, 2026 14:51
bpmct
approved these changes
Oct 23, 2026
doc-check stayed silent on #31159 because "a PR that documents itself needs nothing more" ended the review before anything checked the docs the PR added, and it reported a check it never ran. Of the last 60 non-docs-team PRs it commented on 2. The skill now answers one question: if this change merged now, would the docs on main explain it correctly and clearly? It runs a code-only change model, eight focused personas with a verifier, evidence-or-drop findings, explicit blocking classes, a changed-lines rule for style, and a run trace on every run. Personas find their rules through review-personas front matter in docs/.style, so the style guide can be reorganized without breaking the review. write-docs and AGENTS.md now ask authors to run doc-check on their branch first. The style guide gains rules for leading with the reader's surface and for time-relative words, clarifies Latin abbreviations, adds "as" to the one-meaning table, and DOCS_STYLE_GUIDE.md stops recommending semicolons and "see". Refs DOCS-1410.
…s on unedited paths A first evaluation on #31159 with Solstice-1 Local and Sonnet 5.5 showed both models missing a new section that contradicted an older section of the same page, and Solstice filing a download gap the PR caused as a pre-existing gap. The Continuity Editor now reads the whole page first, coverage and the verifier count gaps on paths the change did not edit, and only rule-anchored findings need a style rule to block. Refs DOCS-1410.
nickvigilante
force-pushed
the
vigilante/docs-1410-doc-check-review-docs-for-accuracy-coverage-audience-and
branch
from
October 23, 2026 16:52
6ecaaf9 to
1ba86a7
Compare
nickvigilante
enabled auto-merge (squash)
October 23, 2026 16:52
nickvigilante
deleted the
vigilante/docs-1410-doc-check-review-docs-for-accuracy-coverage-audience-and
branch
October 23, 2026 17:16
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rebuilds the doc-check skill around one question: if this PR merged now, would the docs on
mainexplain the change correctly and clearly?On #31159 the current skill stayed silent on real defects, because "a PR that documents itself needs nothing more" ended the review before anything checked the docs the PR added.
Changes
output.md,personas/).docs/.style/: each rule page lists its personas inreview-personasfront matter, so the skill finds rules without depending on file paths.asin the one-meaning table. Coverage counts go from 88 to 90..claude/docs/DOCS_STYLE_GUIDE.mdand the old skill no longer recommend semicolons or "see".Out of scope, tracked separately: commit statuses (DOCS-1407), the replay harness (DOCS-923),
draft: true(DOCS-1412), stable rule IDs (DOCS-1413), and "native" wording (DOCS-1409).Testing
make lint/markdown,lint/emdash,lint/style-claims,lint/docs-html,lint/docs-manifest,lint/actions(actionlint and zizmor), and Vale on the changeddocs/.stylefiles all pass."true"is silently ignoredOn #31164, a docs-metadata PR with nothing wrong, v2 reported a clean result with no findings.
v2 fixes the v1 misses: the Continuity Editor reads the whole page first, and gaps on paths the change didn't edit count as caused by the change.
The coverage persona's example comes from #31159, so the v2 rerun on that PR overstates the gain. The replay set in DOCS-923 is the real test.
Risks
Plan and decision log
doc-check redesign: decisions so far
Status: design interview in progress. Nothing implemented.
Tracking: DOCS-918 (parent), DOCS-1407 (commit status), DOCS-925 (outcomes), DOCS-923 (replay), DOCS-926 (harvester), DOCS-1378 (blocking slop), DOCS-597 (renderer class drift).
First principle
If this PR merged now, would the docs on main correctly and clearly explain this exact change?
The requirement runs both ways: every docs claim is grounded in the code, and every behavior that materially changes how a customer uses the product has docs.
"The PR documents itself" is never a reason to stop reviewing.
Scope and audience
Priority and blocking
Order: accuracy, journey completeness, consistency, audience fit, structure, concision, word choice and mechanics.
Blocking findings:
Never blocking:
Evidence and hallucination handling
/doc-check wrong <finding-id> <reason>logs a false finding to Linear and adds the PR to the replay set.Output
Commit status and override (DOCS-1407)
docsstatus on every commit, outcomesfinding,clean,error,skipped.@coder/docsmember overrides to success.@coder/docsafter one business day. The engineer can still merge.Personas
One orchestrator chat with sub-agents, each loading only its rule slice.
IDs match the job; display names are editorial titles.
accuracycoveragelearnerconsistencyaudiencestructurelanguageverifierLanguage and consistency can skip when the PR changes no prose.
The persona list grows as needed.
Rules
docs/.style/is the single human-readable rule source. Fix conflicting text in.claude/docs/DOCS_STYLE_GUIDE.mdand the doc-check skill first (semicolons, unit spacing, "see").review-personasin front matter. Each rule heading carries a stable ID anchor. The bot resolves rules by ID and front matter, never by path.Ethos captured so far
asorsincefor cause (use "because"), niche flags without guide coverage.(e.g., ...)and(for example, ...).Architecture
coder/coderholds the Markdown: skill, personas, rules.coder/doc-checkholds everything else as a hosted GitHub App: orchestration, status, override, re-request, slash commands, verdict parsing, replay harness, persona-to-model map.AGENTS.mdandwrite-docs).@coder/docs, not the whole organization.Models
/doc-check model:opusand in the manual replay.Replay set (DOCS-923)
/doc-check wrongreports, and calibration cases.Calibration set 2 decisions
docs/manifest/unless its front matter hasdraft: true, and block a page that is in the manifest while its front matter hasdraft: true. No page on main usesdraft: trueand the manifest tooling doesn't read it today, so the convention needs defining first.docs/user-guides/, and require that some page covers them.Open
coder-agents-review-botis hosted, to model the doc-check service on it.Overnight decisions (made without Nick; override any of these)
Each entry: the decision, why, and how to undo it.
Linear
draft: true: sub-issue of DOCS-918, Medium, 3 points. Scoped as a deterministic check indocsmanifestgen, not an LLM rule, because blocking rules should be deterministic.styleclaimscheck is its own reviewable change.PR scope
@coder/docs..claude/skills/doc-check/personas/with the agreed names.review-personasfront matter oncontent-guidelines.mdand eight style-guide pages, documented indocs/.style/README.md. Single-line list form so a plaingrepfinds it.editor-setup.mdand the READMEs get none, because they hold no rules.vars.DOC_CHECK_MODEL_CONFIG_ID), so a run never mixes models. Otherwise personas run sequentially.COMMENTreview with inline comments, plus the existing sticky summary comment and marker, because the Slack notice and re-review logic depend on the marker. Findings for agents go in a collapsed YAML block in the summary./doc-check wrongisn't advertised, because it doesn't exist yet.Style guide content
(e.g., ...)and(for example, ...)are both accepted. The comma form is a Don't example but never blocks. Reviewers suggest the English form only when the line changes for another reason.etc.stays accepted in prose, because Ben's position was to accept Latin abbreviations. Nick's personal "tables only" preference is not encoded.asrow added to Choose words that have one meaning, documentation-only, becauseashas too many correct uses for a Vale word list.nowstays allowed for progress inside a sequenced tutorial, because word choice already uses "Now that you added a language".make lint/style-claimsverifies them..claude/docs/DOCS_STYLE_GUIDE.md: removed the semicolon recommendation, "See" as a navigation verb (kept "See which applications", which describes the UI), and a bare150msthreshold (now in code font, matching the content guidelines).docs/or user-facing behavior.Evaluation
adjacent. The Continuity Editor now reads the whole page first, "the change caused it" includes paths the change didn't edit, and only class 4 findings need a style rule to block. Risk: the coverage example mentions downloads, which comes from feat: honor disable-model-invocation in uploaded skill content #31159, so the v2 rerun on feat: honor disable-model-invocation in uploaded skill content #31159 overstates the improvement. The replay set (DOCS-923) is the real test.