Repository navigation
feat: run each doc-check persona as a scoped sub-agent and sweep every style rule - #31649
Closed
nickvigilante wants to merge 10 commits into
Closed
nickvigilante wants to merge 10 commits into
nickvigilante wants to merge 10 commits into
Conversation
…y style rule The orchestrator builds a review packet once, from a copy of the skill and the rule sources at the base commit, and every persona runs as its own sub-agent that reads only the inputs its job needs. The Copy Editor runs as rule sweeps that record a result for every rule section, so a review shows which rules it applied. Each persona has a tool-call budget, sub-agents run at most two at a time by default, and a run that does not finish reports itself as incomplete instead of clean. A dry run mode reviews a pinned head without posting. Refs DOCS-1475
…heck Counting every level-2 and level-3 heading included the example headings inside code fences, so a sweep had to record 216 entries where the rule pages have 183 sections. rule-sections.awk lists only real sections, with line numbers so repeated headings stay distinct. Author self-check now diffs the working tree including new files, and runs markdownlint and Vale on the changed files only, without --fix.
wait_agent watches one agent, so a replay left a finished agent's slot idle for up to 10 minutes while the orchestrator waited on another. The orchestrator now waits 300 seconds on the oldest agent, then lists all agents, and stops an agent by its wall-clock run time.
A replay showed the orchestrator reading every output file in full, including the long checks and rules lists, and re-reading the pages findings cite before the verifiers did the same. It now reads each file's header and findings, and leaves substance to the verifiers. Sub-agent output quotes every string so it parses as YAML, and the quote check uses sed syntax that BSD sed accepts.
…in parts A seeded replay with 68 findings put its review in an attached file and ended without the run trace, and one sweep hit the response token limit writing its output file in one call.
… findings any reviewer raised In the #31159 replay the Analyst wrote the change model only into its output file, so later agents found no change-model.md. The orchestrator now checks for the file and falls back to an empty model. The thread check now covers threads people wrote, as the replay already applied it: three findings matched open human threads and were rightly not repeated.
Three replay orchestrators ran gh auth status as a preflight. For the bot's long installation tokens it masks only the last few dozen characters, so most of the token reached the transcript. The skill now names a read-only API call for the check and forbids gh auth status.
…rule findings The packet's checkout would have detached the author's branch in self-check. Verifier batches in the #31513 replay gave opposite verdicts on the same rule claim for neighboring lines, so findings on one page that cite one rule section now share a batch. A verifier batch whose retry finished no longer marks the run incomplete.
The final #31513 replay posted a blocking finding and, in the same run, recorded that the two style-guide sections it rests on disagree. Conflicts inside one source now get the same treatment as conflicts between sources.
nickvigilante
added a commit
that referenced
this pull request
Nov 12, 2026
…ment (#31994) The doc-check agent posted the raw JSON of its review request as the summary comment on #31385 ([comment](#31385 (comment))), so GitHub showed `{"commit_id": ..., "event": "COMMENT", "body": "## Documentation Check\n..."}` with escaped newlines. The inline review on that run posted correctly. Of the latest summary comments on 42 PRs reviewed since 2026-10-25, this is the only malformed one. This changes `.claude/skills/doc-check/output.md` so the agent: - writes the review request to `review.json` and leaves the review `body` empty, which matches what every other run does; - writes the summary to its own `summary.md`, never reuses `review.json` for it, and posts or edits it with exact `gh api ... -F body=@summary.md` commands; - checks, when it reads the comment back, that the body starts with `## Documentation Check` and ends with the marker, and fixes it with `PATCH` if not. ### Follow-up: workflow guard (not in this PR yet) A workflow job that fails the run when the sticky summary comment does not start with `## Documentation Check` is ready, but the bot that opened this PR can't push changes under `.github/workflows/`. The patch adds a `check-summary-comment` job (`needs: doc-check`, `pull-requests: read`) at the end of `doc-check.yaml`, after `delete-workspace`, so it stays clear of the hunks #31720 changes. It passed `actionlint` (with shellcheck) and `zizmor --offline` with no findings, and a dry run against live comments failed on #31385 and passed on #31896 and #31934. Related open PRs: #31649 (draft) also edits `output.md`, and #31720 edits `doc-check.yaml`. Whichever lands second may need a small rebase. Supersedes #31966, reopened under a human author for attribution. Co-authored-by: coder[bot] <137810501+coder[bot]@users.noreply.github.com>
Collaborator
Author
|
Closing this without merging. On 2026-11-13 the scope of the per-push doc-check review was narrowed to three classes of finding: documentation that is wrong, missing, or broken. The style lenses and the section-by-section rule sweep that this PR extends stop running on each push for now. Mechanical style rules move into Vale as blocking checks, and the rest of the style review is re-evaluated after that. DOCS-1744 is the record of what this PR and #31193 attempted, and of the gaps the narrower scope leaves open. The branch stays in place, because these parts are worth reusing in the redesign:
Generated by Coder Agents on behalf of @nickvigilante. |
nickvigilante
temporarily deployed
to
docs-preview
November 13, 2026 16:42 — with
GitHub Actions
Inactive
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
doc-check now runs each persona as its own sub-agent that reads only what its job needs. The Copy Editor applies every section of every rule source and records a result for each one, and a run limits how much of the shared model it uses.
This PR changes the skill only. The workflow changes come in a separate PR.
Linear: DOCS-1475
Changes
The skill and the rule sources are copied from the base commit, so a PR can't change the rules it is reviewed against.
Together the sweeps cover every
docs/.stylepage with areview-personasfield,.claude/docs/DOCS_STYLE_GUIDE.md, andcontributing/documentation.md.Each sweep records
pass,finding,n/a, ornot_checkedfor each of the 183 rule sections, and the orchestrator checks the count.Each has a tool-call budget and a few named rule sections. A new Analyst persona writes the change model from the code.
incompleteoutcome covers an agent that didn't finish, a sweep that missed a section, or a finding that couldn't be verified. A review never posts after the PR head moves.contract.md,personas/change-model.md,changed-lines.awk,rule-sections.awk, andawk-scripts.test.sh. Onedocs/.style/README.mdparagraph now says which pages the Copy Editor applies.Testing
awk-scripts.test.shpasses under mawk, one-true-awk (the lineage of macOSawk), and GoAWK. Its cases include CRLF line endings, a path with a space,++and--content lines, and nested, indented, and unclosed fences.The sweep-grouping command gives the same output under all three awks.
make lint/emdash, typos, shellcheck, shfmt, and the pre-commit hook pass.Replay results
All replays ran in dry run mode on DeepSeek V4.1 Flash with
max_parallel: 2.Nothing was posted to #31513 or #31159.
7e6475f6c9b07d3580c806c9b07d6c9b07d7e6475f, again5ccc9d1(1) Before
rule-sections.awk, the ledger also counted the example headings inside code fences.What the replays caught
procedural-writing.md#put-the-condition-before-the-instruction.authenticated.WiFi, which is the spellingcontributing/documentation.mdsets. The run flagged it under the style guide's brand-names rule instead of recording a guide conflict, which is 1 false positive.CODER_PG_CONNECTION_URLon a landing page and the idiom "after the fact".iton line 90, which the production run missed.Cost
The budgets are instructions, and agents treat them as soft.
Solstice-1
Three replays on Solstice-1 (all local) ran while about 20 production doc-check reviews on the current skill were active, several with 4 to 7 sub-agents each.
Model steps took 138 seconds at the median, and up to 25 minutes.
None of the three replays spawned a sub-agent in 25 to 125 minutes, so I stopped them. On DeepSeek the same prompts took 5 seconds per step.
The numbers above therefore measure the skill, not Solstice throughput.
Credentials
I scanned 5,332 messages from all seven replays and found no usable credential.
Three orchestrators ran
gh auth status, which printed most of the bot's installation token. Each printed token was truncated and expired within the hour.The skill now forbids
gh auth status.Skill changes the replays caused
Each is its own commit on this branch:
list_agents, so a finished slot is refilled.Plan and decision log
Goal
Every persona runs as its own sub-agent with a small, fixed context, the review applies every rule in the style and contributing guides, and a run stays within what the shared model hardware can serve.
Phase 1 (this plan) changes only the skill. Phase 2 changes the workflow in a separate PR.
Decisions from Nick
Design
Review packet and trust boundary
${TMPDIR:-/tmp}/doc-check/<PR>-<head7>-<UTC timestamp>/:pr.json,review-comments.jsonl,diff.patch,changed-files.txt,changed-lines.txt,related-prs.txt,checks.md,change-model.md,sweeps.txt,manifest.md,out/,batches/.policy/holds the skill fromSKILL_SHAand the rule sources fromPOLICY_SHA(the base branch tip), copied withgit archive. Sub-agents read rules only from there, so a PR cannot change how it is reviewed. A dry run can setSKILL_SHAto a branch.changed-lines.awkturns a zero-context diff intopath:line: text. It handles CRLF, paths with spaces, quoted paths,/dev/null, and++/--content lines.contract.md, and spawns every agent with a fixed five-line prompt. The shared prefix comes first so the model server can reuse its cache.Coverage
docs/.style/*.mdanddocs/.style/style-guide/*.mdpage with areview-personasfield, plus.claude/docs/DOCS_STYLE_GUIDE.mdandcontributing/documentation.md. Onmainthat is 4 sweeps.rule-sections.awklists each level-2 and level-3 heading outside code fences, with its line number. That gives 183 sections onmain. A plain heading count gives 216, because it also counts example headings inside fences.pass,finding,n/a, ornot_checked. The orchestrator compares the count with the script's count, and a short ledger makes the runincomplete.review-personaslists it.change-model.mdfrom the code. For a docs-only change the orchestrator writes an empty model instead.adjacent_gapsand is never a finding.Load and time bounds
max_paralleldefaults to 2 (1 to 4 when the prompt sets it), so the p90 load of 4 reviews needs 8 slots. Verifier batches count against the same limit.list_agentsto refill finished slots. After 90 minutes it sends an agent a stop message, then interrupts it and records itincomplete.unverifiedand is never posted inline.Mechanical checks the orchestrator runs itself
file:line.Outcomes and posting
incompleteoutcome. It posts a summary with an Incomplete section even when no finding survives.review.mdwhen they are long.policy_sha,skill_sha,unverified,max_parallel, anagentslist, andrule_sections.Phase 2 (separate PR)
incompletesummary.github.tokenfrom the workflow instead of the bot token in the workspace.awk-scripts.test.shinto CI.Validation
awk-scripts.test.shunder mawk, one-true-awk, and GoAWK.Not in this PR
.github/workflows/doc-check.yamlstill says to "read everything else from the PR head", which contradicts the rule copy.It also says to "post nothing when no finding survives", which contradicts posting an
incompletesummary.The workflow PR fixes both, posts with
github.token, adds admission control across reviews, and runsawk-scripts.test.shin CI.The ledger proves that every rule section was considered, not that every judgment is right. Smaller sweeps might raise recall, at the cost of more agents. That is untested.
Generated by Coder Agents on behalf of @nickvigilante.