Skip to content

ci: make doc-check reviews smaller and start a new chat per run - #32061

Draft
bpmct wants to merge 1 commit into
mainfrom
bpmct/doc-check-smaller-reviews
Draft

bpmct wants to merge 1 commit into
mainfrom
bpmct/doc-check-smaller-reviews

Conversation

@bpmct

@bpmct bpmct commented Nov 11, 2026

Copy link
Copy Markdown
Member

doc-check reviews are too big and too slow for the local model, and the job result does not match the review result.

From 2026-11-04 to 11-11 (week report):

Changes

Skill (.claude/skills/doc-check/):

  • The 7 personas run in 3 groups: facts (accuracy, coverage), reader (learner, audience, structure), and language (consistency, language). A group loads each rule page once. The persona files keep all their rules.
  • A change with fewer than 150 docs-relevant changed lines runs the groups inline, with no sub-agents.
  • One verifier for 5 or fewer findings. More than that: two verifiers, never more.
  • A style finding that does not change the meaning is a suggestion, never blocking. Blocking stays for wrong facts, missing docs for a new user-facing surface, broken markup, links, and anchors, and failing docs lint. Vale never fails, so a Vale alert is a style finding.
  • A PR in a stack or backport family reads the doc-check comment its siblings got. It keeps the same verdict for the same surface, or says why it differs.
  • The trace gets mode, docs_relevant_lines, groups_run, verifiers, subagents_started, and siblings. The other fields are the same, so old and new runs can be compared.

Workflow (.github/workflows/doc-check.yaml):

  • Each run starts a new chat (force-new-chat) in the PR's existing review workspace (workspace-id). The prompt carries the trace from the previous chat on the PR, so a re-run does not repeat itself.
  • The wait is 5 hours and the job limit is 330 minutes. A chat still running at the end of the wait now ends the job green with a notice and a link to the chat. Other chat errors still fail the job.

Why wait and not end the job when the chat starts: the job has to outlive the chat to stop the workspace when the review ends, and the per-PR concurrency group only keeps two reviews out of one workspace while the job is running. Ending the job early would lose both. The cost is a GitHub-hosted runner that polls for as long as the review takes.

Not tested on live traffic

I checked this with actionlint, zizmor, markdownlint, and a local run of the prompt step with a fake Coder API. It has not run against dogfood. To evaluate it, re-run doc-check on the same 27 PRs from the week report and compare: chats started, time to verdict, job result, and how many of the 17/3/3/4 verdicts change. Also check #31790 and #31849 together, and #31808 for style findings that block.

Overlaps: #31720 changes the same workflow to cancel the run in progress and stop the review chat. #31649 (draft) goes the other way in the skill, with more, narrower sub-agents. Only one of them should merge as is.

Decision log
  • Groups, not new persona files. The persona files stay as they are, so review-personas front matter in docs/.style and the IDs in docs/.style/README.md keep working. Only the Load line and the class rules changed. Rule pages per full review: 113.7k tokens before, 74.6k across the 3 groups, and 42.4k inline.
  • Rule lookup reads front matter only. The old grep -r also matched the example block in docs/.style/README.md.
  • Threshold. Docs-relevant lines are added plus deleted lines, without the no-docs path classes (tests, generated files, lockfiles). The skill has the gh --jq command. 150 is a guess to tune with the evaluation above.
  • The verifier is still a sub-agent in inline mode when findings survive, so the verdict comes from a context that did not write the findings. A clean inline run starts no sub-agents.
  • Consistency findings. A contradiction stays rule. Two terms for one concept is rule only when a reader could take them for two things (the feat(cli): log database migration progress on server startup #31790 case). A UI label that does not match the product is wrong.
  • Posting. A clean run that differs from a sibling's posted verdict posts a summary with only a Sibling PRs section. Every other clean run still posts nothing.
  • Workspace name. The prompt now names doc-check-pr-<number> for a new workspace, so the next run finds it and passes its ID.
  • Risk. A review longer than 5 hours lets the next queued run start a second chat in the same workspace. ci(.github): stop each doc-check review when its run ends #31720's stop script handles that case.

Generated by Coder Agents on behalf of @bpmct.

The skill runs its seven personas in three groups (facts, reader,
language). Each group loads each docs/.style rule page once. A change
under 150 docs-relevant lines runs the groups inline with no
sub-agents. Verification uses one verifier for 5 or fewer findings and
never more than two.

A style finding that leaves the meaning unchanged is a suggestion and
never blocks. A PR in a stack or backport family checks the doc-check
verdict its siblings got and keeps it for the same surface, or says why
it differs.

The workflow starts a new chat for each run in the PR's review
workspace, and passes the previous run's trace in the prompt. It waits
up to 5 hours, and a review that is still running at that point no
longer fails the job.
@bpmct
bpmct temporarily deployed to main November 11, 2026 19:55 — with GitHub Actions Inactive
@bpmct
bpmct temporarily deployed to main November 11, 2026 19:55 — with GitHub Actions Inactive
@linear-code

linear-code Bot commented Nov 13, 2026

Copy link
Copy Markdown

DOCS-1744

bpmct added a commit that referenced this pull request Nov 13, 2026
…run ends (#32185)

## Summary

doc-check reused one chat for every push to a PR, and a review kept
running in Coder after its job stopped waiting for it.
This PR starts a new chat for each run, waits up to 5 hours for it, and
stops the review and its workspace when the run ends for any reason.

On dogfood at 2026-11-13 17:28 UTC, 58 doc-check review chats and 45 of
their sub-agents had status `running`, while 5 workflow runs were in
progress and 1 was queued.
34 of the 58 chats were created before that day, and the oldest on
2026-09-29.
The count covers only chats shared with me, so it is a lower bound.

This combines the workflow half of #32061 with the stop script from
#31720, and changes no skill file.

## Changes

- **New chat per run.**
`force-new-chat` replaces the one chat per PR, so a re-run no longer
carries every earlier turn on the PR.
The prompt gets a `## Previous run` section with the `doc-check-trace`
of the latest finished review.
- **One workspace per PR.**
The job finds `doc-check-pr-<N>`, wakes and starts it, and passes its ID
to the chat.
  On the first run the chat creates the workspace under that name.
- **A started review finishes; pushes are debounced.**
The review job's concurrency group is keyed on the PR URL with
`cancel-in-progress: false`. GitHub keeps one pending run per group and
replaces it on each push, so the next review covers the latest head.
A `debounce` job sleeps 5 minutes on `synchronize`, in its own group
with `cancel-in-progress: true`, so a burst of pushes starts one review.
Other triggers do not wait.
Merging the PR still cancels a running review (the `delete-workspace`
job keeps `cancel-in-progress: true`), then deletes the workspace.
- **Every run stops its review.**
`.github/scripts/doc-check-stop-review.sh` runs before the review and in
a final `always()` step.
It drops queued messages, interrupts the chat and its sub-agents, and
stops the workspace.
- **5-hour wait.**
`wait-timeout-seconds` goes from 2400 to 18000, and the job timeout from
50 to 330 minutes.
The job now stays in its concurrency group for the whole review, so the
next run for the PR waits for it.
- **Previous trace lookup.**
`.github/scripts/doc-check-previous-trace.sh` reads the 5 newest review
chats for the PR and prints the first trace with an `outcome:` line.
It reads more than one chat because a cancelled run's chat has no trace.
  A failed lookup costs only that context and never fails the job.
- The `GITHUB_OUTPUT` delimiter for the prompt is random, because the
trace is model output.

### Behavior changes to check

| Before | After | Why |
|--------|-------|-----|
| A review past the wait ended the job with a green notice and kept
running (#32097) | The last step stops the review, and the job fails |
Nothing watches a review after the job ends |
| A review kept running after its PR merged | The merge cancels the run
| The result has no reader |
| The `doc-check` label forced one review, on the `labeled` event | The
label forces a review on every push until it is removed | The push run
would otherwise skip on paths with no docs surface |
| Runs for one PR queued | A started review finishes; pushes during it
collapse into one pending run for the latest head, after a 5-minute
debounce | Cancelling threw away prefill and starved busy PRs of a
verdict |

## Testing

Static checks, all passing: `actionlint`, `zizmor`, `shellcheck
--external-sources`, and `shfmt -d`.

Script tests, run locally:

- `doc-check-stop-review.test.sh`: 48 cases against a fake API.
- `doc-check-previous-trace.test.sh`: 22 cases, including CRLF in the
trace text and in the PR URL.
- Mutation check on the trace script: 25 one-line breaks, and the suite
failed on 24.
  The survivor was a redundant trim, which I removed.
- The trace extraction ran against 12 real finished doc-check chats on
dogfood and returned a trace for all 12 (750 to 3,500 characters).

End to end on dogfood, on this PR, 2026-11-13 UTC:

| Run | Trigger | Result |
|-----|---------|--------|
| [1](https://github.com/coder/coder/actions/runs/43474029515) |
dispatch, 17:33 | No workspace and no earlier chat. New chat created the
workspace `doc-check-pr-32185` and had 2 running sub-agents at 17:48:23.
|
| [2](https://github.com/coder/coder/actions/runs/43476429715) |
dispatch, 17:48:25 | Cancelled run 1 at 17:48:27. Run 1's last step
interrupted the chat and both sub-agents and stopped the workspace by
17:48:51. Run 2 then started the same workspace, opened a second chat in
it, finished clean at 17:52:54, and stopped the workspace. |
| [3](https://github.com/coder/coder/actions/runs/43477511866) |
dispatch, 17:55:08 | Cancelled by run 4 while the workspace was
starting. Its last step stopped the workspace mid-start. No chat was
created. |
| [4](https://github.com/coder/coder/actions/runs/43477572348) |
`ready_for_review`, 17:55:31 | Waited for run 3 to end, then opened a
third chat in the same workspace. That chat's first message carries run
2's trace under `## Previous run`. |

Not covered:

- The 5-hour timeout path has not run live.
The fake-API tests cover the script, and the summary text is a few lines
of shell.
- The merge path (`delete-workspace` cancelling a review) has not run
live.
- No CI job runs the two `.test.sh` suites.
The workflow runs on `ubuntu-latest` only, and the scripts were not run
on macOS or Windows.

## Follow-ups, not in this PR

- Skip a run when the head SHA or the PR diff matches the last reviewed
one.
- Set a status on the head SHA from the trace.
- Cancel a review when its PR closes without merging.

Supersedes #31720 and the workflow half of #32061.

Linear: DOCS-1745

<details>
<summary>Plan and decision log</summary>

This is PR 1 of a 5-part stack from the design doc "Design: reliable,
low-cost doc-check on Solstice".

1. Workflow base: new chat per run, previous trace, workspace ID, 5-hour
wait, cancel-in-progress, and the stop script. **This PR.**
2. Skip in Actions: same head SHA, same PR diff, and a delta since the
last reviewed SHA with no docs surface.
3. Visible outcome: a status on the head SHA from the trace, for clean,
findings, error, and skipped.
4. Packet and lint in Actions: PR data, diffs, open threads, and
deterministic check results, written once.
5. Skill: scope to wrong, missing, and broken, tiers, a sub-agent cap,
and one long wait. Gated by an eval.

Decisions in this PR:

- **Why this first.**
A re-run's first prompt was 1.01 times the previous run's last prompt,
so each push re-sent the whole history of the PR.
A new chat per run removes that, and stopping stale reviews removes load
that no job is watching.
- **5 hours, not 25 or 40 minutes.**
Reviews on the local model took a median of 61 minutes when clean and
247 minutes with findings (2026-11-04 to 11-11).
  A shorter wait would stop most reviews with findings before they post.
With `cancel-in-progress`, a new push still ends the review of the old
head at once.
- **A timeout fails the job.**
  This reverses #32097, where a timeout was a notice.
  A green job with no review reads as a clean result.
- **The stop runs twice.**
The first step catches a review left by a run whose last step did not
finish, for example a runner that died.
- **The trace lookup is a script with tests, not inline YAML.**
It parses model output, and it has to handle chats with no trace, a
trace with no outcome, and CRLF.
- **The trace is passed as a record, not as instructions.**
  The prompt says so, and asks for a review of the whole PR.
A delta review needs the skip logic and the ledger from later PRs in the
stack.
- **No skill change.**
  The skill changes are gated by an eval that does not exist yet.

</details>

Follow-up commit `64a09655a1` (finish-then-rerun and debounce) generated
by Coder Agents on behalf of @bpmct.

_Generated by Coder Agents on behalf of @nickvigilante._

---------

Co-authored-by: Ben Potter <me@bpmct.net>

This branch was previously deployed

1 inactive deployment
main — 305671a1 Deployed Nov 11, 2026 by bpmct via community-label #254485
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant