Repository navigation
Conversation
The skill runs its seven personas in three groups (facts, reader, language). Each group loads each docs/.style rule page once. A change under 150 docs-relevant lines runs the groups inline with no sub-agents. Verification uses one verifier for 5 or fewer findings and never more than two. A style finding that leaves the meaning unchanged is a suggestion and never blocks. A PR in a stack or backport family checks the doc-check verdict its siblings got and keeps it for the same surface, or says why it differs. The workflow starts a new chat for each run in the PR's review workspace, and passes the previous run's trace in the prompt. It waits up to 5 hours, and a review that is still running at that point no longer fails the job.
bpmct
added a commit
that referenced
this pull request
Nov 13, 2026
…run ends (#32185) ## Summary doc-check reused one chat for every push to a PR, and a review kept running in Coder after its job stopped waiting for it. This PR starts a new chat for each run, waits up to 5 hours for it, and stops the review and its workspace when the run ends for any reason. On dogfood at 2026-11-13 17:28 UTC, 58 doc-check review chats and 45 of their sub-agents had status `running`, while 5 workflow runs were in progress and 1 was queued. 34 of the 58 chats were created before that day, and the oldest on 2026-09-29. The count covers only chats shared with me, so it is a lower bound. This combines the workflow half of #32061 with the stop script from #31720, and changes no skill file. ## Changes - **New chat per run.** `force-new-chat` replaces the one chat per PR, so a re-run no longer carries every earlier turn on the PR. The prompt gets a `## Previous run` section with the `doc-check-trace` of the latest finished review. - **One workspace per PR.** The job finds `doc-check-pr-<N>`, wakes and starts it, and passes its ID to the chat. On the first run the chat creates the workspace under that name. - **A started review finishes; pushes are debounced.** The review job's concurrency group is keyed on the PR URL with `cancel-in-progress: false`. GitHub keeps one pending run per group and replaces it on each push, so the next review covers the latest head. A `debounce` job sleeps 5 minutes on `synchronize`, in its own group with `cancel-in-progress: true`, so a burst of pushes starts one review. Other triggers do not wait. Merging the PR still cancels a running review (the `delete-workspace` job keeps `cancel-in-progress: true`), then deletes the workspace. - **Every run stops its review.** `.github/scripts/doc-check-stop-review.sh` runs before the review and in a final `always()` step. It drops queued messages, interrupts the chat and its sub-agents, and stops the workspace. - **5-hour wait.** `wait-timeout-seconds` goes from 2400 to 18000, and the job timeout from 50 to 330 minutes. The job now stays in its concurrency group for the whole review, so the next run for the PR waits for it. - **Previous trace lookup.** `.github/scripts/doc-check-previous-trace.sh` reads the 5 newest review chats for the PR and prints the first trace with an `outcome:` line. It reads more than one chat because a cancelled run's chat has no trace. A failed lookup costs only that context and never fails the job. - The `GITHUB_OUTPUT` delimiter for the prompt is random, because the trace is model output. ### Behavior changes to check | Before | After | Why | |--------|-------|-----| | A review past the wait ended the job with a green notice and kept running (#32097) | The last step stops the review, and the job fails | Nothing watches a review after the job ends | | A review kept running after its PR merged | The merge cancels the run | The result has no reader | | The `doc-check` label forced one review, on the `labeled` event | The label forces a review on every push until it is removed | The push run would otherwise skip on paths with no docs surface | | Runs for one PR queued | A started review finishes; pushes during it collapse into one pending run for the latest head, after a 5-minute debounce | Cancelling threw away prefill and starved busy PRs of a verdict | ## Testing Static checks, all passing: `actionlint`, `zizmor`, `shellcheck --external-sources`, and `shfmt -d`. Script tests, run locally: - `doc-check-stop-review.test.sh`: 48 cases against a fake API. - `doc-check-previous-trace.test.sh`: 22 cases, including CRLF in the trace text and in the PR URL. - Mutation check on the trace script: 25 one-line breaks, and the suite failed on 24. The survivor was a redundant trim, which I removed. - The trace extraction ran against 12 real finished doc-check chats on dogfood and returned a trace for all 12 (750 to 3,500 characters). End to end on dogfood, on this PR, 2026-11-13 UTC: | Run | Trigger | Result | |-----|---------|--------| | [1](https://github.com/coder/coder/actions/runs/43474029515) | dispatch, 17:33 | No workspace and no earlier chat. New chat created the workspace `doc-check-pr-32185` and had 2 running sub-agents at 17:48:23. | | [2](https://github.com/coder/coder/actions/runs/43476429715) | dispatch, 17:48:25 | Cancelled run 1 at 17:48:27. Run 1's last step interrupted the chat and both sub-agents and stopped the workspace by 17:48:51. Run 2 then started the same workspace, opened a second chat in it, finished clean at 17:52:54, and stopped the workspace. | | [3](https://github.com/coder/coder/actions/runs/43477511866) | dispatch, 17:55:08 | Cancelled by run 4 while the workspace was starting. Its last step stopped the workspace mid-start. No chat was created. | | [4](https://github.com/coder/coder/actions/runs/43477572348) | `ready_for_review`, 17:55:31 | Waited for run 3 to end, then opened a third chat in the same workspace. That chat's first message carries run 2's trace under `## Previous run`. | Not covered: - The 5-hour timeout path has not run live. The fake-API tests cover the script, and the summary text is a few lines of shell. - The merge path (`delete-workspace` cancelling a review) has not run live. - No CI job runs the two `.test.sh` suites. The workflow runs on `ubuntu-latest` only, and the scripts were not run on macOS or Windows. ## Follow-ups, not in this PR - Skip a run when the head SHA or the PR diff matches the last reviewed one. - Set a status on the head SHA from the trace. - Cancel a review when its PR closes without merging. Supersedes #31720 and the workflow half of #32061. Linear: DOCS-1745 <details> <summary>Plan and decision log</summary> This is PR 1 of a 5-part stack from the design doc "Design: reliable, low-cost doc-check on Solstice". 1. Workflow base: new chat per run, previous trace, workspace ID, 5-hour wait, cancel-in-progress, and the stop script. **This PR.** 2. Skip in Actions: same head SHA, same PR diff, and a delta since the last reviewed SHA with no docs surface. 3. Visible outcome: a status on the head SHA from the trace, for clean, findings, error, and skipped. 4. Packet and lint in Actions: PR data, diffs, open threads, and deterministic check results, written once. 5. Skill: scope to wrong, missing, and broken, tiers, a sub-agent cap, and one long wait. Gated by an eval. Decisions in this PR: - **Why this first.** A re-run's first prompt was 1.01 times the previous run's last prompt, so each push re-sent the whole history of the PR. A new chat per run removes that, and stopping stale reviews removes load that no job is watching. - **5 hours, not 25 or 40 minutes.** Reviews on the local model took a median of 61 minutes when clean and 247 minutes with findings (2026-11-04 to 11-11). A shorter wait would stop most reviews with findings before they post. With `cancel-in-progress`, a new push still ends the review of the old head at once. - **A timeout fails the job.** This reverses #32097, where a timeout was a notice. A green job with no review reads as a clean result. - **The stop runs twice.** The first step catches a review left by a run whose last step did not finish, for example a runner that died. - **The trace lookup is a script with tests, not inline YAML.** It parses model output, and it has to handle chats with no trace, a trace with no outcome, and CRLF. - **The trace is passed as a record, not as instructions.** The prompt says so, and asks for a review of the whole PR. A delta review needs the skip logic and the ledger from later PRs in the stack. - **No skill change.** The skill changes are gated by an eval that does not exist yet. </details> Follow-up commit `64a09655a1` (finish-then-rerun and debounce) generated by Coder Agents on behalf of @bpmct. _Generated by Coder Agents on behalf of @nickvigilante._ --------- Co-authored-by: Ben Potter <me@bpmct.net>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
doc-check reviews are too big and too slow for the local model, and the job result does not match the review result.
From 2026-11-04 to 11-11 (week report):
docs/.stylepages, but the pages are only 42.4k tokens without repeats.Changes
Skill (
.claude/skills/doc-check/):facts(accuracy, coverage),reader(learner, audience, structure), andlanguage(consistency, language). A group loads each rule page once. The persona files keep all their rules.mode,docs_relevant_lines,groups_run,verifiers,subagents_started, andsiblings. The other fields are the same, so old and new runs can be compared.Workflow (
.github/workflows/doc-check.yaml):force-new-chat) in the PR's existing review workspace (workspace-id). The prompt carries the trace from the previous chat on the PR, so a re-run does not repeat itself.Why wait and not end the job when the chat starts: the job has to outlive the chat to stop the workspace when the review ends, and the per-PR concurrency group only keeps two reviews out of one workspace while the job is running. Ending the job early would lose both. The cost is a GitHub-hosted runner that polls for as long as the review takes.
Not tested on live traffic
I checked this with actionlint, zizmor, markdownlint, and a local run of the prompt step with a fake Coder API. It has not run against dogfood. To evaluate it, re-run doc-check on the same 27 PRs from the week report and compare: chats started, time to verdict, job result, and how many of the 17/3/3/4 verdicts change. Also check #31790 and #31849 together, and #31808 for style findings that block.
Overlaps: #31720 changes the same workflow to cancel the run in progress and stop the review chat. #31649 (draft) goes the other way in the skill, with more, narrower sub-agents. Only one of them should merge as is.
Decision log
review-personasfront matter indocs/.styleand the IDs indocs/.style/README.mdkeep working. Only theLoadline and the class rules changed. Rule pages per full review: 113.7k tokens before, 74.6k across the 3 groups, and 42.4k inline.grep -ralso matched the example block indocs/.style/README.md.gh --jqcommand. 150 is a guess to tune with the evaluation above.rule. Two terms for one concept isruleonly when a reader could take them for two things (the feat(cli): log database migration progress on server startup #31790 case). A UI label that does not match the product iswrong.doc-check-pr-<number>for a new workspace, so the next run finds it and passes its ID.Generated by Coder Agents on behalf of @bpmct.