Skip to content

feat: run each doc-check persona as a scoped sub-agent and sweep every style rule - #31649

Closed
nickvigilante wants to merge 10 commits into
mainfrom
vigilante/docs-1475-doc-check-scope-each-personas-sub-agent-sweep-every-style
Closed

nickvigilante wants to merge 10 commits into
mainfrom
vigilante/docs-1475-doc-check-scope-each-personas-sub-agent-sweep-every-style

Conversation

@nickvigilante

Copy link
Copy Markdown
Collaborator

Summary

doc-check now runs each persona as its own sub-agent that reads only what its job needs. The Copy Editor applies every section of every rule source and records a result for each one, and a run limits how much of the shared model it uses.
This PR changes the skill only. The workflow changes come in a separate PR.

Linear: DOCS-1475

Changes

  • Review packet. The orchestrator collects the diff, changed lines, PR data, review threads, CI results, and change model once, in a directory unique to the run, and sub-agents read from it.
    The skill and the rule sources are copied from the base commit, so a PR can't change the rules it is reviewed against.
  • Every rule, section by section. The Copy Editor runs as about four sweeps of at most 8,000 words of rules each.
    Together the sweeps cover every docs/.style page with a review-personas field, .claude/docs/DOCS_STYLE_GUIDE.md, and contributing/documentation.md.
    Each sweep records pass, finding, n/a, or not_checked for each of the 183 rule sections, and the orchestrator checks the count.
  • Narrow lenses. The other personas cover what a line-by-line sweep can't see: the code, other pages, page shape, the reader, and what a newcomer learns.
    Each has a tool-call budget and a few named rule sections. A new Analyst persona writes the change model from the code.
  • Bounded load. At most 2 sub-agents run at once, or 1 to 4 when the prompt sets it. An agent still running after 90 minutes is stopped, and verifier batches hold at most 6 findings.
  • Honest outcomes. A new incomplete outcome covers an agent that didn't finish, a sweep that missed a section, or a finding that couldn't be verified. A review never posts after the PR head moves.
  • Dry run mode. It reviews a pinned head with a skill from a branch, and posts nothing.
  • New files: contract.md, personas/change-model.md, changed-lines.awk, rule-sections.awk, and awk-scripts.test.sh. One docs/.style/README.md paragraph now says which pages the Copy Editor applies.

Testing

  • awk-scripts.test.sh passes under mawk, one-true-awk (the lineage of macOS awk), and GoAWK. Its cases include CRLF line endings, a path with a space, ++ and -- content lines, and nested, indented, and unclosed fences.
    The sweep-grouping command gives the same output under all three awks.
  • markdownlint, the table formatter, Vale, make lint/emdash, typos, shellcheck, shfmt, and the pre-commit hook pass.
  • An author self-check of this branch with the new skill found 2 blocking issues in the README sentence and 9 gaps in the self-check instructions. All are fixed.
  • Four dry-run replays through Coder Agents posted nothing to GitHub. The results are below.
Replay results

All replays ran in dry run mode on DeepSeek V4.1 Flash with max_parallel: 2.
Nothing was posted to #31513 or #31159.

Replay Skill Wall time First sub-agent Sub-agents Peak running Rule ledger Posted (blocking, suggestion) Rejected by verifier
#31513 at 7e6475f 6c9b07d 82 min 2.4 min 15 3, briefly 213 of 216 (1) 12, 5 9
#31159 at 3580c80 6c9b07d 123 min 1.5 min 17 2 216 of 216 (1) 11, 4 2
Page with 63 planted violations 6c9b07d 94 min 2.1 min 27 2 216 of 216 (1) 63, 5 15, and 17 merged
#31513 at 7e6475f, again 5ccc9d1 79 min 1.9 min 18 3, briefly 184 of 184 10, 6 11

(1) Before rule-sections.awk, the ledger also counted the example headings inside code fences.

What the replays caught

  • Planted violations: 58 of 60 valid plants. These cover every rule source: content guidelines, every style-guide page, DOCS_STYLE_GUIDE, contributing, accuracy, structure, and audience.
    • It missed procedural-writing.md#put-the-condition-before-the-instruction.
    • It also missed the DOCS_STYLE_GUIDE rule to lead with the environment variable, on a sentence the run deleted as wrong for another reason.
    • Three plants were my own mistakes. The verifier rejected two of them with code evidence: the dashboard really does share at authenticated.
    • The third was WiFi, which is the spelling contributing/documentation.md sets. The run flagged it under the style guide's brand-names rule instead of recording a guide conflict, which is 1 false positive.
  • feat: honor disable-model-invocation in uploaded skill content #31159: all 7 findings from the feat: make doc-check review docs quality, not only missing docs #31193 evaluation.
    • 3 were posted and 1 went to the trace.
    • The other 3 were raised and then dropped, correctly, because open review threads already raise them.
    • 9 more verified findings, such as the 8-skill limit and chips that disappear when the organization changes.
  • docs: add quick paths to the install landing page and retire the CLI page #31513: compared with the 8 findings of an earlier production run on the old skill.
    • Both replays found 5 of the 8.
    • Both missed CODER_PG_CONNECTION_URL on a landing page and the idiom "after the fact".
    • One replay found the table-versus-phase contradiction in a different form.
    • Each replay found 11 or 12 other verified issues, including "Day-two operations" and an unclear it on line 90, which the production run missed.

Cost

  • Personas used 14 to 84 tool calls against budgets of 15 to 30. Verifiers used 15 to 83 against 15, and tool output was 36 to 260 kB per agent.
    The budgets are instructions, and agents treat them as soft.
  • The longest sub-agent took 24 minutes.
  • The orchestrator used 96 to 200 tool calls. After the change that limits what it reads, it used 102 calls and 207 kB on docs: add quick paths to the install landing page and retire the CLI page #31513, compared with 96 calls and 249 kB before.

Solstice-1

Three replays on Solstice-1 (all local) ran while about 20 production doc-check reviews on the current skill were active, several with 4 to 7 sub-agents each.
Model steps took 138 seconds at the median, and up to 25 minutes.
None of the three replays spawned a sub-agent in 25 to 125 minutes, so I stopped them. On DeepSeek the same prompts took 5 seconds per step.
The numbers above therefore measure the skill, not Solstice throughput.

Credentials

I scanned 5,332 messages from all seven replays and found no usable credential.
Three orchestrators ran gh auth status, which printed most of the bot's installation token. Each printed token was truncated and expired within the hour.
The skill now forbids gh auth status.

Skill changes the replays caused

Each is its own commit on this branch:

  • A rule-section count that skips code fences.
  • Polling with list_agents, so a finished slot is refilled.
  • An orchestrator that reads only output headers and findings, and doesn't re-judge.
  • Quoted YAML strings, and long output files written in parts.
  • The trace always comes last.
  • A fixed path for the change model.
  • Dropping findings that any open review thread already raises.
  • A GitHub access check that doesn't print the token.
  • A self-check that leaves the author's branch alone.
  • Same-rule findings verified together.
  • No blocking on a finding that two rules disagree about.
Plan and decision log

Goal

Every persona runs as its own sub-agent with a small, fixed context, the review applies every rule in the style and contributing guides, and a run stays within what the shared model hardware can serve.
Phase 1 (this plan) changes only the skill. Phase 2 changes the workflow in a separate PR.

Decisions from Nick

  1. Concurrency: whatever handles the load best, avoids timeouts, and works with the hardware.
  2. Coverage: doc-check must catch everything the style and contributing guides set, more thoroughly than a human review.
  3. Phasing: skill-only PR first, workflow PR second.
  4. Tracking: a new DOCS issue (DOCS-1475), not DOCS-879.

Design

Review packet and trust boundary

  • The orchestrator builds one packet per run at ${TMPDIR:-/tmp}/doc-check/<PR>-<head7>-<UTC timestamp>/: pr.json, review-comments.jsonl, diff.patch, changed-files.txt, changed-lines.txt, related-prs.txt, checks.md, change-model.md, sweeps.txt, manifest.md, out/, batches/.
  • policy/ holds the skill from SKILL_SHA and the rule sources from POLICY_SHA (the base branch tip), copied with git archive. Sub-agents read rules only from there, so a PR cannot change how it is reviewed. A dry run can set SKILL_SHA to a branch.
  • changed-lines.awk turns a zero-context diff into path:line: text. It handles CRLF, paths with spaces, quoted paths, /dev/null, and ++/-- content lines.
  • The orchestrator never reads persona files, rule pages, or contract.md, and spawns every agent with a fixed five-line prompt. The shared prefix comes first so the model server can reuse its cache.

Coverage

  • The Copy Editor runs as sweeps: rule pages bin-packed into groups of at most 8,000 words. The groups are every docs/.style/*.md and docs/.style/style-guide/*.md page with a review-personas field, plus .claude/docs/DOCS_STYLE_GUIDE.md and contributing/documentation.md. On main that is 4 sweeps.
  • rule-sections.awk lists each level-2 and level-3 heading outside code fences, with its line number. That gives 183 sections on main. A plain heading count gives 216, because it also counts example headings inside fences.
  • Each sweep records one ledger entry per section: pass, finding, n/a, or not_checked. The orchestrator compares the count with the script's count, and a short ledger makes the run incomplete.
  • The lens personas are narrowed to what the sweeps cannot see: code (Fact Checker, Cartographer), other sentences and pages (Continuity Editor), page shape and mechanics (Architect), the page's reader (Reader Advocate), and what a reader learns (Newcomer). Each names its few rule sections by heading and finds them on pages whose review-personas lists it.
  • A new Analyst persona writes change-model.md from the code. For a docs-only change the orchestrator writes an empty model instead.
  • Precedence when sources disagree: content guidelines, then style guide, then DOCS_STYLE_GUIDE.md, then contributing/documentation.md. Repository facts override all of them. A conflict goes to adjacent_gaps and is never a finding.

Load and time bounds

  • Hardware facts used: the GB10 handles about 8 concurrent requests comfortably. Historic concurrent doc-check reviews ran at p50 1 and p90 4.
  • max_parallel defaults to 2 (1 to 4 when the prompt sets it), so the p90 load of 4 reviews needs 8 slots. Verifier batches count against the same limit.
  • The orchestrator waits on the oldest running agent for 300 seconds, then calls list_agents to refill finished slots. After 90 minutes it sends an agent a stop message, then interrupts it and records it incomplete.
  • Per-persona tool-call budgets, plus about 100 kB of tool output per agent. The platform does not enforce these.
  • Verifier batches hold at most 6 findings, as whole groups. A batch that ends without finishing is retried once, split in two. A finding still not verified is unverified and is never posted inline.

Mechanical checks the orchestrator runs itself

  • The quote exists at file:line.
  • No review thread, from doc-check or a person, already raises the finding.
  • No CI annotation on the same line refers to the same rule anchor.
  • Findings with the same file, line, and rule are merged.
  • The orchestrator does not judge substance. That is the verifier's job.

Outcomes and posting

  • A new incomplete outcome. It posts a summary with an Incomplete section even when no finding survives.
  • Before posting, the orchestrator reads the PR head again and posts nothing if the head moved.
  • Dry run mode posts nothing. Its final message holds the would-be comments, the summary, and the trace, or a path to review.md when they are long.
  • The trace gains policy_sha, skill_sha, unverified, max_parallel, an agents list, and rule_sections.

Phase 2 (separate PR)

  • Workflow prompt: stop saying "read everything else from the PR head", and allow posting an incomplete summary.
  • Post with github.token from the workflow instead of the bot token in the workspace.
  • Admission control across concurrent reviews, and wait timeouts that match observed run times.
  • Wire awk-scripts.test.sh into CI.

Validation

Not in this PR

  • Workflow (phase 2). The prompt in .github/workflows/doc-check.yaml still says to "read everything else from the PR head", which contradicts the rule copy.
    It also says to "post nothing when no finding survives", which contradicts posting an incomplete summary.
    The workflow PR fixes both, posts with github.token, adds admission control across reviews, and runs awk-scripts.test.sh in CI.
  • Run-to-run variance. Two replays of docs: add quick paths to the install landing page and retire the CLI page #31513 found the same 5 production findings and different extras.
    The ledger proves that every rule section was considered, not that every judgment is right. Smaller sweeps might raise recall, at the cost of more agents. That is untested.
  • Untested. Windows Git Bash (gawk) for the awk scripts, and the new skill on Solstice under normal load.

Generated by Coder Agents on behalf of @nickvigilante.

…y style rule

The orchestrator builds a review packet once, from a copy of the skill and
the rule sources at the base commit, and every persona runs as its own
sub-agent that reads only the inputs its job needs. The Copy Editor runs as
rule sweeps that record a result for every rule section, so a review shows
which rules it applied. Each persona has a tool-call budget, sub-agents run
at most two at a time by default, and a run that does not finish reports
itself as incomplete instead of clean. A dry run mode reviews a pinned head
without posting.

Refs DOCS-1475
…heck

Counting every level-2 and level-3 heading included the example headings
inside code fences, so a sweep had to record 216 entries where the rule
pages have 183 sections. rule-sections.awk lists only real sections, with
line numbers so repeated headings stay distinct.

Author self-check now diffs the working tree including new files, and runs
markdownlint and Vale on the changed files only, without --fix.
wait_agent watches one agent, so a replay left a finished agent's slot idle for up to 10 minutes while the orchestrator waited on another. The orchestrator now waits 300 seconds on the oldest agent, then lists all agents, and stops an agent by its wall-clock run time.
A replay showed the orchestrator reading every output file in full, including the long checks and rules lists, and re-reading the pages findings cite before the verifiers did the same. It now reads each file's header and findings, and leaves substance to the verifiers. Sub-agent output quotes every string so it parses as YAML, and the quote check uses sed syntax that BSD sed accepts.
…in parts

A seeded replay with 68 findings put its review in an attached file and ended without the run trace, and one sweep hit the response token limit writing its output file in one call.
… findings any reviewer raised

In the #31159 replay the Analyst wrote the change model only into its output file, so later agents found no change-model.md. The orchestrator now checks for the file and falls back to an empty model. The thread check now covers threads people wrote, as the replay already applied it: three findings matched open human threads and were rightly not repeated.
Three replay orchestrators ran gh auth status as a preflight. For the bot's long installation tokens it masks only the last few dozen characters, so most of the token reached the transcript. The skill now names a read-only API call for the check and forbids gh auth status.
…rule findings

The packet's checkout would have detached the author's branch in self-check. Verifier batches in the #31513 replay gave opposite verdicts on the same rule claim for neighboring lines, so findings on one page that cite one rule section now share a batch. A verifier batch whose retry finished no longer marks the run incomplete.
The final #31513 replay posted a blocking finding and, in the same run, recorded that the two style-guide sections it rests on disagree. Conflicts inside one source now get the same treatment as conflicts between sources.
@linear-code

linear-code Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

DOCS-1475

DOCS-1744

nickvigilante added a commit that referenced this pull request Nov 12, 2026
…ment (#31994)

The doc-check agent posted the raw JSON of its review request as the
summary comment on #31385
([comment](#31385 (comment))),
so GitHub showed `{"commit_id": ..., "event": "COMMENT", "body": "##
Documentation Check\n..."}` with escaped newlines. The inline review on
that run posted correctly. Of the latest summary comments on 42 PRs
reviewed since 2026-10-25, this is the only malformed one.

This changes `.claude/skills/doc-check/output.md` so the agent:

- writes the review request to `review.json` and leaves the review
`body` empty, which matches what every other run does;
- writes the summary to its own `summary.md`, never reuses `review.json`
for it, and posts or edits it with exact `gh api ... -F
body=@summary.md` commands;
- checks, when it reads the comment back, that the body starts with `##
Documentation Check` and ends with the marker, and fixes it with `PATCH`
if not.

### Follow-up: workflow guard (not in this PR yet)

A workflow job that fails the run when the sticky summary comment does
not start with `## Documentation Check` is ready, but the bot that
opened this PR can't push changes under `.github/workflows/`. The patch
adds a `check-summary-comment` job (`needs: doc-check`, `pull-requests:
read`) at the end of `doc-check.yaml`, after `delete-workspace`, so it
stays clear of the hunks #31720 changes. It passed `actionlint` (with
shellcheck) and `zizmor --offline` with no findings, and a dry run
against live comments failed on #31385 and passed on #31896 and #31934.

Related open PRs: #31649 (draft) also edits `output.md`, and #31720
edits `doc-check.yaml`. Whichever lands second may need a small rebase.


Supersedes #31966, reopened under a human author for attribution.

Co-authored-by: coder[bot] <137810501+coder[bot]@users.noreply.github.com>
@nickvigilante

Copy link
Copy Markdown
Collaborator Author

Closing this without merging.

On 2026-11-13 the scope of the per-push doc-check review was narrowed to three classes of finding: documentation that is wrong, missing, or broken. The style lenses and the section-by-section rule sweep that this PR extends stop running on each push for now. Mechanical style rules move into Vale as blocking checks, and the rest of the style review is re-evaluated after that.

DOCS-1744 is the record of what this PR and #31193 attempted, and of the gaps the narrower scope leaves open.

The branch stays in place, because these parts are worth reusing in the redesign:

  • the dry-run mode, which reviews a pinned head and posts nothing
  • the review packet, collected once per run and read by sub-agents
  • the incomplete outcome
  • changed-lines.awk and rule-sections.awk, with their tests
  • the planted-violations page, as an eval case

Generated by Coder Agents on behalf of @nickvigilante.

@nickvigilante
nickvigilante temporarily deployed to docs-preview November 13, 2026 16:42 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 16:42 — with GitHub Actions Inactive
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 13, 2026

This branch was previously deployed

2 inactive deployments
docs-preview — 03edea6a Deployed Nov 13, 2026 by nickvigilante via cleanup #45
main — 03edea6a Deployed Nov 13, 2026 by nickvigilante via cla #256751
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant