Repository navigation
ci(.github): stop each doc-check review when its run ends - #31720
nickvigilante wants to merge 9 commits into
Conversation
doc-check reviews on the local model took a median of 150 minutes, because doc-check alone kept about 40 review chats active and each model step slowed from about 20 seconds to over 200. Most of that load was wasted: a review kept running after its job stopped waiting, the next push queued another full review behind it, and a workspace left running past an hour held an expired GitHub token, so 159 of 244 reviews ended on a 401. A new push now cancels the PR's run in progress. A new script stops the PR's review chat and its sub-agents, drops their queued messages, and stops the review workspace. The job runs it before the review, so the review starts on an idle chat and a freshly started workspace, and after every run, including a cancelled one. The wait drops to 25 minutes, and a review past it is stopped and fails the job. Refs DOCS-1478
|
/coder-agents-review |
|
Chat: Review posted | View chat Review history
deep-review v0.13.0 | Round 9 | Last posted: Round 9, 116 findings (4 P2, 17 P3, 70 P4, 25 Nit), COMMENT. Review Finding inventoryFinding inventory: PR #31720Findings
Contested and acknowledgedCRF-4 (P4, doc-check-stop-review.sh:147) - CR strip handles input that cannot occur
CRF-6 (P4, doc-check-stop-review.sh:174) - Exact-name workspace lookup duplicated in three places
CRF-12 (P3, doc-check-stop-review.sh:62) - paused chat reported stopped
CRF-14 (P3, doc-check.yaml:490) - No bound on concurrent reviews across PRs
CRF-18 (P4, doc-check-stop-review.sh:178) - Prepare restarts only the workspace found by name
CRF-68 (P3, doc-check.yaml:505) - 1500-second wait stops long reviews
Law analysisR9 (head f038096): effective LOC reported +1102 -34 (Law: the 704-line test file is counted as production; about 430 production lines). Verdict: Don't split. Enforcement: Advisory. One reviewable idea (each doc-check run stops its own review when it ends); the cancel and label changes are safe only with the post-run stop step. Round logRound 1Netero first pass: 1 P3, 5 P4, 1 Nit, 1 Note. No P0, panel proceeds. Law not run (effective additions 522). Round 2 updateBLOCKED. CRF-14 deferred without a ticket ("planned as a follow-up", no issue linked). No review. Churn guard: 24 addressed, 1 acknowledged (CRF-18), 2 contested (CRF-4, CRF-6), 1 deferred without ticket (CRF-14). Status cells unchanged until the panel runs. Head: 5a270bc. Round 3Churn guard PROCEED: CRF-14 now tracked in DOCS-1480. Head unchanged (5a270bc). Round type: fix round. Since the last panel round (R1, 38970da) no files were added or renamed, no PR goal changed, effective additions grew 117; Round 4 updateBLOCKED. CRF-10 silent: the R3 re-raise (runs tested 38970da) has no author reply, its thread was resolved without a response, and Testing still cites only runs 41729412601 and 41731313388 with no tested SHA. No review. Churn guard: 15 of 16 open findings addressed in bf1aa8b (CRF-48, 50 to 63), status cells unchanged until the panel verifies. Head: bf1aa8b. Round 5Churn guard PROCEED: CRF-10 answered (runs 41742809326 and 41743065015, head_sha bf1aa8b confirmed via the public GitHub API; Testing names the SHA and lists uncovered paths). All 16 open findings addressed. Round type: fix round. Since the last panel round (R3, 5a270bc) no files added or renamed, no goal change, effective additions grew 118; Round 6Churn guard PROCEED: 15 addressed in ccecdfa, CRF-68 acknowledged (author took the first proposed option). Round type: fix round (no files added or renamed, no goal change, +102 effective additions since R5; changes trace to R5 findings). Panel: Knov via its R5 agent (CRF-68 dispute check-in); new agents for Meruem and Takumi (CRF-67 phase deadline), Gon and Leorio (comments and messages changed), Chopper (CRF-72), Mafu-san (CRF-73 description), Bisky (CRF-74 to 76 tests), Mafuuu (CRF-77), Melody (CRF-78 queue loop). Netero R6: no findings, CRF-66 verified. Wildcard (random): Zoro. Round 7Churn guard PROCEED: 14 of 14 addressed in af47edc (stop_chat replaced by stop_chats, has_time and nap removed, sleep_if_time added). Round type: fix round: no files added or renamed, no goal change, +19 effective additions since R6; the stop_chats rewrite is the fix CRF-83 proposed. Panel (new agents, no contested findings): Knov, Meruem, Bisky (CRF-83), Mafu-san, Chopper, Mafuuu (CRF-84), Gon, Leorio, Zoro (CRF-86, 89 to 95), Takumi (CRF-87, 88). Netero R7: CRF-98 P4. Wildcard (random): Kite. Round 8Churn guard PROCEED: 16 of 16 addressed in 10eb134 (stop_chats and stop_workspace replaced by run_phase, sweep_chat, sweep_workspace; the test fake gains a virtual clock, fail_seq, and queue_on_interrupt). Inventory validation: no non-posted item's assumptions changed (CRF-97's poll-count backstop is gone with Round 9Churn guard PROCEED: 17 of 17 addressed in f038096 (failed stops retried through ws_stop sent/failed, clear_queue continues after a failed delete, one busy status set, failed-interrupt warning removed, fake answers 409 during builds and has a virtual-clock hang guard, new cases). Inventory validation: CRF-127 stays dropped (the labels are unchanged); CRF-132 is a body note on unchanged yaml. Round type: fix round: no files added or renamed, no goal change, +141 effective additions since R8; each change traces to a round 8 finding. Law triage: effective additions 1102 exceed 1000 and grew 580 since round 1 (522), and Law has never run, so Law runs. About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
This PR adds a script that stops a PR's doc-check chat, sub-agents, and workspace before and after each run, sets cancel-in-progress: true, and cuts the wait to 25 minutes. Findings: 2 P2, 7 P3, 13 P4, 6 Nit.
Out of scope (needs a ticket or explicit acceptance by a human):
.github/workflows/doc-check.yaml:215: the dispatch URL check accepts anyowner/repo, but the workspace name uses only the PR number, so a dispatch forother/repo/pull/123starts, and now also stops, the workspace of coder/coder PR 123..github/workflows/test-*.sh: no CI job runs the repository's shell test scripts, including this PR's, so each one stops matching the code it tests without anyone seeing a failure.coderd/x/chatd/agentadmission.go:12: chatd's only concurrent-agent cap is deployment-wide and fixed (5 root, 10 sub-agent slots, or off), so a deployment cannot cap one model or one bot user.coder-workflow-bottemplate (not in this repository): the workspace mints its GitHub installation token once at startup, so any session past one hour gets 401s..github/workflows/doc-check.yaml:352: the workspace start wait repeats the same 180-second budget inline (seq 36,sleep 5), so a change to the script's budget leaves this one unchanged..github/workflows/doc-check.yaml:342: Prepare the review workspace starts the workspace only forstop/stoppedand prints "Workspace state is usable." for a failed stop or start build, so the review runs on a workspace whose last build failed.
Notes:
- The
test-go-pgandtest-go-pg-17failures are not caused by this diff;.github/actions/test-go-pg/action.yamlreferences none of the changed files. - The cleanup step's worst case (3 interrupt rounds of 60 s plus a 180 s workspace wait, about 360 s) is longer than the 5 minutes GitHub gives
always()steps after a cancel. The newer run's Stop the previous review repeats the stop, so the result is a force-terminated step in the cancelled run's log. - The job now runs a checked-out script with
DOC_CHECK_CODER_SESSION_TOKEN. For same-repositorypull_requestruns the PR author already controls the workflow file at the same ref, so this adds no new capability.
🤖 This review was automatically generated with Coder Agents.
Key the concurrency group on the PR URL, so a manual run and a pull request run for one PR cancel each other instead of stopping each other's review. Keep forcing a review while the PR has the doc-check label, because a push now cancels the labeled run. In the script, interrupt again on the next poll, re-read chats after dropping their queues, warn on every failed request, give each request a 30-second limit, and treat only a stopped build as stopped: a failed start gets a stop, and a failed stop warns. Bound both cleanup steps, and say in the step summary when a review is stopped at the wait. Cover each warning branch, failed builds, and paused chats in the test, and keep transitions in the fake API for two reads, so the cases show that the script waits for them. Refs DOCS-1478
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review of 5a270bc. The author replied to every round 1 finding, and the commit addresses most of them. One finding is deferred with no ticket, so this round's review is blocked until it is resolved:
- No bound on concurrent reviews across PRs (
.github/workflows/doc-check.yaml:496). The reply says the cap is planned as a follow-up, once the post-PR concurrency is measured, but no issue is linked; DOCS-1478 is this PR's own ticket. This needs a human decision: link a ticket for the cap, or state that shipping without one is accepted, given that a burst of PRs past the model's slow-down point gives zero finished reviews.
The panel reviews the new commit, including the two contested findings and the acknowledged one, after this is resolved.
🤖 This review was automatically generated with Coder Agents.
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review of 5a270bc, which responds to round 1. Most round 1 findings are fixed, and reviewers confirmed the fixes in the code and tests. The carriage-return strip, the shared workspace lookup, the workspace restart by name, and leaving paused out of any_active are accepted as defended, and the concurrency cap is tracked in DOCS-1480. The end-to-end testing finding is reopened, because both dogfood runs ran 38970da, not this head. New findings: 3 P3, 9 P4, 3 Nit.
Out of scope (needs a ticket or explicit acceptance by a human):
.github/workflows/doc-check.yaml:304: the exact-namedoc-check-pr-<N>lookup exists in Prepare the review workspace, delete-workspace, anddoc-check-stop-review.sh, so a rename that misses one copy makes that lookup find no workspace..github/workflows/doc-check.yaml:220: the dispatch URL check accepts anyowner/repo, but the prompt hard-codescoder/coderand the workspace name uses only the PR number, so a dispatch for another repository reviews the wrong PR..github/workflows/doc-check.yaml:50: adoc-checklabel added to a merged PR passes the jobifand runs a review; if delete-workspace already deleteddoc-check-pr-<N>, the review creates a new one that no later event deletes.
🤖 This review was automatically generated with Coder Agents.
The stop script now ends within one time budget (STOP_BUDGET_SECONDS, 270 seconds by default) instead of per-loop sleep counts, and caps each request at the time left. It also stops a workspace whose earlier stop failed, re-reads a chat's queue after each delete, and looks up the doc-check-pr-<number> workspace only for this repository's pull requests. The workflow comments and summaries now match: any run in the group cancels the one in progress, the label reviews every later push, the timeout summary no longer repeats the wait or claims the stop worked, and both cleanup steps say what their 6-minute limit covers. Refs DOCS-1478
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review of bf1aa8b. The commit responds to every round 3 code finding except one. This round's review is blocked until that one gets a response:
- The end-to-end runs in Testing ran 38970da, not the code under review (
.github/workflows/doc-check.yaml:56). Its thread was resolved with no reply, and Testing still cites only runs 41729412601 and 41731313388 without naming the tested commit. Since then, 5a270bc and bf1aa8b rewrotestop_chatandstop_workspace, added the time budget, and moved the concurrency key to the PR URL; none of that has run on GitHub or the deployed Coder, and a manual run overlapping a pull request run has not run at any commit. Response needed: repeat the cancel and timeout runs at the current head (plus one manual run overlapping a pull request run) and name the SHA in Testing, or state in Testing that the runs tested 38970da and list what they do not cover.
The panel reviews bf1aa8b after this is resolved.
🤖 This review was automatically generated with Coder Agents.
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review of bf1aa8b, which responds to round 3. Reviewers checked the round 3 fixes in the code and tests and found them in place, and the end-to-end runs now ran this head (both report head_sha bf1aa8b). New findings: 2 P3, 11 P4, 3 Nit. The two P3s are about time: slow Coder requests during the chat phase can use the whole budget so the workspace is never stopped, and the 25-minute wait stops reviews that the decision log's own estimate says take longer even without load.
🤖 This review was automatically generated with Coder Agents.
The chats now get a deadline at a third of the budget, enforced on every request, queue read, and poll, so slow chat requests no longer use the time the workspace stop needs. Every workspace gets at least one read, and sleeps are cut to the time left. The script also reads deleted workspaces, drops a full queue of 20 without a false warning, says when it could not read a chat or ran out of time instead of claiming the chat keeps running, and words the workspace warnings for the states that reach them. New cases cover two matching reviews, a full queue, a deleted workspace, a mixed-case repository URL, and the request time limit and its floor. Refs DOCS-1478
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review of ccecdfa, which responds to round 5. Reviewers checked the round 5 fixes in the code and tests and found them in place: per-phase deadlines, the first workspace read after the deadline, include_deleted, the full-queue read, and the new cases. The 25-minute wait tradeoff is accepted now that the decision log states the count. New findings: 1 P3, 10 P4, 3 Nit. Most are about the new deadline code: a regression where a second review chat is never interrupted, branches with no test, and comments and warnings that describe the deadline incorrectly.
🤖 This review was automatically generated with Coder Agents.
Every review chat now gets its queue dropped and a first interrupt before the script waits on any of them, so a first chat that uses up the chat phase no longer leaves a second one running. The chat phase's deadline limits only the waiting. A poll now starts only when a full poll interval and the read after it fit in the phase, so sleeps are never cut short and each poll reads after it sleeps. The phase boundaries sit together in main under a named constant, the warnings say what a failed queue drop leads to, and the start-after-stop warning names the likely cause. Refs DOCS-1478
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review of af47edc, which responds to round 6. Reviewers checked the round 6 fixes and found them in place: every review chat gets its queue dropped and a first interrupt before any wait, sleep_if_time gates the polls in both loops, the round 6 header, comment, and warning fixes are in, and the two end-to-end runs in Testing ran this head. New findings: 1 P2, 1 P3, 12 P4, 2 Nit. The P2 is the rewritten chat-budget case, which fails on unchanged code under load, so the Testing sentence about 8 copies does not hold.
Three of the new findings are branches that a one-line mutation removes with the suite still passing; rounds 1, 5, and 6 had findings of the same kind. Deleting or inverting each new condition and running the suite before pushing would find these.
🤖 This review was automatically generated with Coder Agents.
Each phase, chats then workspaces, now runs one sweep function on every item before it waits on any, then sweeps the items that have not settled until the phase ends. Workspaces no longer wait on each other, every sweep drops a busy chat's queue before it interrupts, and a chat or workspace whose read fails is read again instead of dropped. The tests now run on a virtual clock that the fake sleep, date, and slow requests share, so the timing cases are deterministic and take no real time. The fake can fail a path on chosen calls and queue a message when a chat is interrupted, and new cases cover a read that fails and then recovers, a read lost after a success, two workspaces, and the poll rule. Refs DOCS-1478
|
/coder-agents-review |
There was a problem hiding this comment.
Re-review of 10eb134, which responds to round 7 by replacing the two stop loops with run_phase and per-item sweeps and giving the test fake a virtual clock. Reviewers confirmed the round 7 fixes: chat-budget no longer depends on machine speed (8 copies of the suite passed at load average 134), every chat's queue is dropped before its first interrupt, workspaces no longer wait on each other, the poll counts are gone, the new read-recovery cases fail under the breaks they target, and the two end-to-end runs in Testing ran this head. New findings: 1 P2, 2 P3, 10 P4, 4 Nit.
This is the third round in a row where a Testing claim did not match what reviewers measured (the 8-copy claim in round 7, the mutation claim now). Listing only the checks that were run, with their results, would let a reader rely on the section.
Notes:
.github/workflows/doc-check.yaml:506: until the reviews started before this PR end, the PR's own timeout run made 6 model steps in 25 minutes against a median of 29, so most runs right after merge may post nothing. Running the stop script once for each open PR with a doc-check chat at merge time would remove that load; a human can decide whether that is worth doing.
🤖 This review was automatically generated with Coder Agents.
A failed workspace stop request is now retried after a new read instead of settling the workspace, so a 409 from a build that started after the read, or a transient error, no longer leaves the workspace running. The warning moves to the end of the phase, and a build that starts after a failed request is reported as a build that did not finish. A failed interrupt no longer warns, because a chat that ends its turn after the read answers 409 and the next sweep reads it again. A queue delete that fails is retried after a new read, because chatd answers 404 for a message it promoted after the read. The busy statuses are written once, and the after-stop checks are one block. The fake now answers a stop with 409 while a build is in progress, and can drop a queued message after a read. New cases cover a failed queue read, a queue past 20 deletes, a promoted message, a failed interrupt, a failed or retried stop, each build status, and a workspace read lost after a success. The suite stops a case that polls past 3000 virtual seconds, so its real-time limit is only a last resort. Refs DOCS-1478
|
/coder-agents-review |
|
@nickvigilante ⛔ This review has reached its per-chat spend limit ($106.48 / $100.00). Further review rounds are paused. To raise the limit and continue, comment: This is a per-chat budget, separate from any account-level usage limit.
|
|
/coder-agents-review set-spend-limit:250 |
There was a problem hiding this comment.
Re-review of f038096, which responds to round 8. Reviewers confirmed the round 8 fixes with targeted cases and mutations: failed stops are retried, the fake answers a stop during a build with an error and each in-progress status has a case, the busy status set is written once, clear_queue moves past a promoted message, the hang guard runs on the virtual clock (two-workspaces took 10 seconds on one core), two reviewers reran 20 and 6 of the 88 breaks in the Testing table and each failed its case, and the two end-to-end runs ran this head. New findings: 1 P3, 5 P4, 4 Nit. Most are about what the step log shows when a request fails repeatedly or a stop's result is unknown.
Notes:
.github/scripts/doc-check-stop-review.sh:109: swappingreport_workspace's first two branches, or moving theqid == failedcheck below the last-passbreak, passes the suite; each changes only which warning prints, and Testing says the 88 breaks are not every edit..github/workflows/doc-check.yaml:68: the review job anddelete-workspaceshare a concurrency group, so withcancel-in-progress: truea review job that starts after a merge (from thedoc-checklabel or a dispatch) cancels a running delete. This needs the merged-PR review that the out-of-scope label item already allows, and fixing that item also fixes this.
🤖 This review was automatically generated with Coder Agents.
Addresses the round 9 coder-agents-review findings on #31720. - Warn once per chat about its queue. GitHub keeps only the first 10 warning annotations of a step, and a busy chat's queue is read again on every sweep, so repeats hid the warnings that a review may still run. - Warn "Could not interrupt chat <id>" for a chat that still reads as running after its last interrupt failed, instead of reporting it as a chat that ignored interrupts. - Count a stop build that follows a failed stop request as this run's stop, because a request that timed out may have been accepted. Print the workspace's status when a failed request is followed by a stopped workspace, so the log no longer ends at "Stopping workspace". - Drop the queue of a chat in error. chatd keeps it, and the next message sent to the chat runs the oldest queued one first. - Make the fake API fail a workspace ID it has no entry for, as coderd does, and fix the test comments the review flagged. DOCS-1478
…ment (#31994) The doc-check agent posted the raw JSON of its review request as the summary comment on #31385 ([comment](#31385 (comment))), so GitHub showed `{"commit_id": ..., "event": "COMMENT", "body": "## Documentation Check\n..."}` with escaped newlines. The inline review on that run posted correctly. Of the latest summary comments on 42 PRs reviewed since 2026-10-25, this is the only malformed one. This changes `.claude/skills/doc-check/output.md` so the agent: - writes the review request to `review.json` and leaves the review `body` empty, which matches what every other run does; - writes the summary to its own `summary.md`, never reuses `review.json` for it, and posts or edits it with exact `gh api ... -F body=@summary.md` commands; - checks, when it reads the comment back, that the body starts with `## Documentation Check` and ends with the marker, and fixes it with `PATCH` if not. ### Follow-up: workflow guard (not in this PR yet) A workflow job that fails the run when the sticky summary comment does not start with `## Documentation Check` is ready, but the bot that opened this PR can't push changes under `.github/workflows/`. The patch adds a `check-summary-comment` job (`needs: doc-check`, `pull-requests: read`) at the end of `doc-check.yaml`, after `delete-workspace`, so it stays clear of the hunks #31720 changes. It passed `actionlint` (with shellcheck) and `zizmor --offline` with no findings, and a dry run against live comments failed on #31385 and passed on #31896 and #31934. Related open PRs: #31649 (draft) also edits `output.md`, and #31720 edits `doc-check.yaml`. Whichever lands second may need a small rebase. Supersedes #31966, reopened under a human author for attribution. Co-authored-by: coder[bot] <137810501+coder[bot]@users.noreply.github.com>
…run ends (#32185) ## Summary doc-check reused one chat for every push to a PR, and a review kept running in Coder after its job stopped waiting for it. This PR starts a new chat for each run, waits up to 5 hours for it, and stops the review and its workspace when the run ends for any reason. On dogfood at 2026-11-13 17:28 UTC, 58 doc-check review chats and 45 of their sub-agents had status `running`, while 5 workflow runs were in progress and 1 was queued. 34 of the 58 chats were created before that day, and the oldest on 2026-09-29. The count covers only chats shared with me, so it is a lower bound. This combines the workflow half of #32061 with the stop script from #31720, and changes no skill file. ## Changes - **New chat per run.** `force-new-chat` replaces the one chat per PR, so a re-run no longer carries every earlier turn on the PR. The prompt gets a `## Previous run` section with the `doc-check-trace` of the latest finished review. - **One workspace per PR.** The job finds `doc-check-pr-<N>`, wakes and starts it, and passes its ID to the chat. On the first run the chat creates the workspace under that name. - **A started review finishes; pushes are debounced.** The review job's concurrency group is keyed on the PR URL with `cancel-in-progress: false`. GitHub keeps one pending run per group and replaces it on each push, so the next review covers the latest head. A `debounce` job sleeps 5 minutes on `synchronize`, in its own group with `cancel-in-progress: true`, so a burst of pushes starts one review. Other triggers do not wait. Merging the PR still cancels a running review (the `delete-workspace` job keeps `cancel-in-progress: true`), then deletes the workspace. - **Every run stops its review.** `.github/scripts/doc-check-stop-review.sh` runs before the review and in a final `always()` step. It drops queued messages, interrupts the chat and its sub-agents, and stops the workspace. - **5-hour wait.** `wait-timeout-seconds` goes from 2400 to 18000, and the job timeout from 50 to 330 minutes. The job now stays in its concurrency group for the whole review, so the next run for the PR waits for it. - **Previous trace lookup.** `.github/scripts/doc-check-previous-trace.sh` reads the 5 newest review chats for the PR and prints the first trace with an `outcome:` line. It reads more than one chat because a cancelled run's chat has no trace. A failed lookup costs only that context and never fails the job. - The `GITHUB_OUTPUT` delimiter for the prompt is random, because the trace is model output. ### Behavior changes to check | Before | After | Why | |--------|-------|-----| | A review past the wait ended the job with a green notice and kept running (#32097) | The last step stops the review, and the job fails | Nothing watches a review after the job ends | | A review kept running after its PR merged | The merge cancels the run | The result has no reader | | The `doc-check` label forced one review, on the `labeled` event | The label forces a review on every push until it is removed | The push run would otherwise skip on paths with no docs surface | | Runs for one PR queued | A started review finishes; pushes during it collapse into one pending run for the latest head, after a 5-minute debounce | Cancelling threw away prefill and starved busy PRs of a verdict | ## Testing Static checks, all passing: `actionlint`, `zizmor`, `shellcheck --external-sources`, and `shfmt -d`. Script tests, run locally: - `doc-check-stop-review.test.sh`: 48 cases against a fake API. - `doc-check-previous-trace.test.sh`: 22 cases, including CRLF in the trace text and in the PR URL. - Mutation check on the trace script: 25 one-line breaks, and the suite failed on 24. The survivor was a redundant trim, which I removed. - The trace extraction ran against 12 real finished doc-check chats on dogfood and returned a trace for all 12 (750 to 3,500 characters). End to end on dogfood, on this PR, 2026-11-13 UTC: | Run | Trigger | Result | |-----|---------|--------| | [1](https://github.com/coder/coder/actions/runs/43474029515) | dispatch, 17:33 | No workspace and no earlier chat. New chat created the workspace `doc-check-pr-32185` and had 2 running sub-agents at 17:48:23. | | [2](https://github.com/coder/coder/actions/runs/43476429715) | dispatch, 17:48:25 | Cancelled run 1 at 17:48:27. Run 1's last step interrupted the chat and both sub-agents and stopped the workspace by 17:48:51. Run 2 then started the same workspace, opened a second chat in it, finished clean at 17:52:54, and stopped the workspace. | | [3](https://github.com/coder/coder/actions/runs/43477511866) | dispatch, 17:55:08 | Cancelled by run 4 while the workspace was starting. Its last step stopped the workspace mid-start. No chat was created. | | [4](https://github.com/coder/coder/actions/runs/43477572348) | `ready_for_review`, 17:55:31 | Waited for run 3 to end, then opened a third chat in the same workspace. That chat's first message carries run 2's trace under `## Previous run`. | Not covered: - The 5-hour timeout path has not run live. The fake-API tests cover the script, and the summary text is a few lines of shell. - The merge path (`delete-workspace` cancelling a review) has not run live. - No CI job runs the two `.test.sh` suites. The workflow runs on `ubuntu-latest` only, and the scripts were not run on macOS or Windows. ## Follow-ups, not in this PR - Skip a run when the head SHA or the PR diff matches the last reviewed one. - Set a status on the head SHA from the trace. - Cancel a review when its PR closes without merging. Supersedes #31720 and the workflow half of #32061. Linear: DOCS-1745 <details> <summary>Plan and decision log</summary> This is PR 1 of a 5-part stack from the design doc "Design: reliable, low-cost doc-check on Solstice". 1. Workflow base: new chat per run, previous trace, workspace ID, 5-hour wait, cancel-in-progress, and the stop script. **This PR.** 2. Skip in Actions: same head SHA, same PR diff, and a delta since the last reviewed SHA with no docs surface. 3. Visible outcome: a status on the head SHA from the trace, for clean, findings, error, and skipped. 4. Packet and lint in Actions: PR data, diffs, open threads, and deterministic check results, written once. 5. Skill: scope to wrong, missing, and broken, tiers, a sub-agent cap, and one long wait. Gated by an eval. Decisions in this PR: - **Why this first.** A re-run's first prompt was 1.01 times the previous run's last prompt, so each push re-sent the whole history of the PR. A new chat per run removes that, and stopping stale reviews removes load that no job is watching. - **5 hours, not 25 or 40 minutes.** Reviews on the local model took a median of 61 minutes when clean and 247 minutes with findings (2026-11-04 to 11-11). A shorter wait would stop most reviews with findings before they post. With `cancel-in-progress`, a new push still ends the review of the old head at once. - **A timeout fails the job.** This reverses #32097, where a timeout was a notice. A green job with no review reads as a clean result. - **The stop runs twice.** The first step catches a review left by a run whose last step did not finish, for example a runner that died. - **The trace lookup is a script with tests, not inline YAML.** It parses model output, and it has to handle chats with no trace, a trace with no outcome, and CRLF. - **The trace is passed as a record, not as instructions.** The prompt says so, and asks for a review of the whole PR. A delta review needs the skip logic and the ledger from later PRs in the stack. - **No skill change.** The skill changes are gated by an eval that does not exist yet. </details> Follow-up commit `64a09655a1` (finish-then-rerun and debounce) generated by Coder Agents on behalf of @bpmct. _Generated by Coder Agents on behalf of @nickvigilante._ --------- Co-authored-by: Ben Potter <me@bpmct.net>
Summary
doc-check reviews on the local Solstice model take a median of 150 minutes, so most runs time out.
The skill isn't the bottleneck: at an unloaded step time, the same reviews would take a median of 13 minutes.
The model is slow because doc-check alone keeps a median of 40 review chats active, and most of that work is wasted:
This PR makes each run stop its review when the run ends, so active reviews don't outnumber running jobs unless a stop fails, and every review starts with a fresh token. The stop steps warn when a stop fails.
Linear: DOCS-1478
Changes
cancel-in-progress: true: a new push cancels the PR's run in progress instead of queueing behind it.New
.github/scripts/doc-check-stop-review.shruns two phases, each with one sweep function per kind of item:A failed interrupt or queue delete is tried again after a new read, because chatd answers 409 or 404 when a turn ends between the read and the request.
It also drops the queue of a chat in
error, because the next message sent to that chat runs the oldest queued one first.A queue failure warns once per chat, because GitHub keeps only the first 10 warning annotations of a step, and a chat that is still running after its last interrupt failed gets its own warning.
doc-check-pr-<N>): a sweep waits out a build in progress and sends a stop to any other workspace that isn't stopped, including one whose earlier stop failed.A failed stop request is sent again on the next sweep, because coderd answers 409 when a build started after the read.
A stop build after a failed request counts as this run's stop, because a request that timed out may have been accepted.
Each phase sweeps every item once, then sweeps the ones that haven't settled until its deadline: the chats get the first third of
STOP_BUDGET_SECONDS(270 seconds by default), and the workspaces the rest.Each request gets the time left in its phase, at least 1 second and at most 30. The deadline decides whether a poll starts, not when its sweep ends, so a slow deployment can push the script past the budget.
It looks up
doc-check-pr-<N>only for this repository's PRs, because the name has only the number.The job runs the script before the review (Stop the previous review and its workspace) and after every run, including a cancelled one (Stop the review and its workspace,
if: always()).The existing step that starts a stopped workspace then brings it up fresh, so the startup script mints a new installation token.
The concurrency group is keyed on the PR URL, so any new run for a PR, including a manual one, cancels the run in progress instead of stopping its review mid-run.
A PR with the
doc-checklabel gets a review on every push until the label is removed, because a push now cancels the labeled run in progress. The skip summary and the file header say so.The wait drops from 40 to 25 minutes, and
timeout-minutesfrom 50 to 45. A review past the wait is stopped, the job fails, and the step summary says the job tried to stop it.The old Stop the review workspace step is gone. It needed the action's
workspace-idoutput, which the action doesn't set on a timeout, so it never ran for the reviews that most needed it.Testing
End to end on dogfood, with
workflow_dispatchruns of this branch against this PR. At f038096, the head before 41f8f18:waiting5 seconds later, and stoppeddoc-check-pr-31720(04:42:19 to 04:42:51).gh run cancelat 04:43:57 stopped the chat and the workspace the same way (04:44:10 to 04:44:26).At bf1aa8b, an earlier head, runs 41742809326 and 41743065015 showed the same cancel and queueing results, plus:
gh pr diffandgh pr viewcalls returned the PR.waiting5 seconds later, and stopped the workspace (00:23:21 to 00:23:32).A timeout runs the same script call as a cancel, and the later commits change only the script and workflow comments, so the 25-minute timeout run wasn't repeated.
The timeout run's review made 6 model steps in 25 minutes, at a median of 339 seconds a step, because 20 other doc-check reviews from before this PR were still running on the model (40 active chats counting sub-agents, at 23:59).
Not covered end to end:
html_urlishttps://github.com/coder/coder/pull/31720, the formpr_urlis validated to.error. Only the script test covers them.The timeout path at 41f8f18 ran on this PR's
ready_for_reviewrun (42681464271): the wait ended at 1500 seconds (19:39:51 UTC) and the job failed. The last step interrupted the chat, which the next poll read as stopped 5 seconds later, and stoppeddoc-check-pr-31720(19:39:52 to 19:40:09).bash .github/scripts/doc-check-stop-review.test.shruns the real script against a fake Coder API: a fakecurl,sleep, anddateonPATH, sharing a virtual clock that slow fake requests also advance. No case waits in real time, and a case fails if it polls past 3000 virtual seconds. It has 48 cases.They cover running and idle reviews with sub-agents and queued messages, two matching reviews, a first review whose slow queue read uses up the chat phase, a full queue of 20 and one of 21, messages queued during an interruption, a message chatd promotes between the queue read and the delete,
requires_actionand paused chats, a chat inerrorwith queued messages, a chat that ignores interrupts, two workspaces where one build never ends, each build status in progress (the fake answers a stop with 409 during one), a failed start, a stop build that failed in this run or an earlier one, a stop request that times out after coderd accepts it, a failed stop request on a workspace another build stopped, a deleted workspace, the poll rule and the request limits (30-second cap, time left in the phase, 1-second floor), the exact workspace name match, another repository's PR, a mixed-case repository URL, and a PR URL with a trailing CR.For each of the 8 requests the script makes, at least one case fails it: the chat and workspace lookups, the chat and workspace reads (failing always, then recovering, and after a success), the queue read and delete, the interrupt (failing once and always), and the stop (failing always, once, before a build starts, and after coderd accepts it).
At f038096, I ran the first 88 one-line breaks in the table below against the suite, and each failed at least one case.
Two of them failed by crashing on an unset variable, which
set -uwould also do in production.At 41f8f18, I ran the last 6, one for each behavior fix in that commit, and each failed its new case.
The breaks don't cover every possible edit.
At 41f8f18, on this 128-core workspace, 8 copies of the suite at once passed in 18 seconds, 4 copies pinned to 2 cores (
taskset -c 0,1) in 25, and 8 copies pinned to 1 core in 108.No CI job runs shell tests, so this runs by hand on Linux, like the existing
.github/workflows/test-*.shscripts. The script and the test need bash 4.4 or later, whichubuntu-latesthas.At 41f8f18, shfmt, shellcheck,
make lint/actions/actionlint, andmake lint/actions/zizmorpass, with no zizmor findings.Mutation check: 94 one-line breaks, all caught
apirunning-reviewapichat-budgetapi--max-timerunning-reviewsleep_if_timebuild-never-endssleep_if_time>to>=build-never-endssleep_if_timestuck-chatchat_treerunning-reviewclear_queuerunning-reviewclear_queuerunning-reviewclear_queue<=to<in the loop boundfull-queueclear_queuequeue-read-failsclear_queuequeue-read-failsclear_queuebreakinstead ofreturnon an empty queuerunning-reviewclear_queuequeue-delete-failsclear_queuequeue-promotedclear_queuequeue-delete-failsclear_queuequeue-delete-failsclear_queuequeue-overflowclear_queuequeue-delete-failsclear_queuequeue-promotedclear_queuequeue-overflowsweep_chatchat-read-failssweep_chatstuck-chatsweep_chatrunningfrom the busy statusesrunning-reviewsweep_chatrequires_actionfrom the busy statusesrequires-actionsweep_chatinterruptingfrom the busy statusesrunning-reviewsweep_chatpausedfrom the busy statusespaused-chatsweep_chatrunning-reviewsweep_chatclear_queuerunning-reviewsweep_chatidle-reviewsweep_chatidle-reviewsweep_chatrunning-reviewsweep_chatrequires_actionrequires-actionsweep_chatinterruptingrunning-reviewsweep_chatrequires-actionsweep_chatstuck-chatsweep_chatstuck-chatsweep_chatinterrupt-failssweep_chatinterrupt-failssweep_chatrunning-reviewreport_chatchat-read-failsreport_chatstuck-chatsweep_workspaceworkspace-read-failssweep_workspacebuild-never-endssweep_workspaceworkspace-read-lostsweep_workspaceinclude_deleted=truebuild-deletingsweep_workspacependingbuild-pendingsweep_workspacestartingbuild-startingsweep_workspacestoppingrunning-reviewsweep_workspacecancelingbuild-cancelingsweep_workspacedeletingbuild-deletingsweep_workspacestop-fails-then-buildsweep_workspacerunning-reviewsweep_workspaceidle-reviewsweep_workspacebuild-deletingsweep_workspaceidle-reviewsweep_workspacerunning-reviewsweep_workspacefailed-stopsweep_workspacerunning-reviewsweep_workspacefailed-stopsweep_workspacerunning-after-stopsweep_workspacefailed-stopsweep_workspacerunning-reviewsweep_workspacestop-failssweep_workspacerunning-reviewsweep_workspacerunning-reviewreport_workspaceworkspace-read-failsreport_workspacebuild-never-endsreport_workspacestop-failsreport_workspacebuild-never-endsrun_phaserunning-reviewrun_phaseidle-reviewrun_phaserunning-reviewrun_phaserunning-reviewrun_phaseidle-reviewrun_phasestuck-chatmainchat-budgetmainchat-budgetmaintwo-reviewsmaindoc-check-pr-<N>for any repositoryother-repositorymainmixed-case-repositorymainexact-workspace-namemainPR_URLcrlf-pr-urlmaintwo-workspacesmainbudget-caps-requestsmainworkspace-lookup-failsmain/pull/URLsrunning-reviewmainrunning-reviewclear_queuequeue-read-failssweep_chatclear_queuefor a chat inerrorerror-chatssweep_chatinterrupt-never-succeedssweep_workspacestop-fails-then-stoppedsweep_workspacestop-times-out-after-applymainworkspace_idtwo-reviewsUnrelated CI flake
At af47edc,
test-go-pg (macos-latest)failed incliTestUpdateTemplateVersionApplyFailure/true(workspace_template_version_test.go:319: expected "partial start state", got ""). At 10eb134,Storybookfailed inoauth2appssettingspageview--update-error-outranks-stale-load-error(play function could not find its error text). This PR changes no Go or frontend code. I re-ran each failed job, and both passed.Decision log
Measurements: every
doc-check-botchat shared with the organization since Oct 30 17:54 UTC, read fromGET /api/v2/chatsand each chat's messages: 244 parent review turns and 468 sub-agent turns, each timed from its user message to its last message.Most of the tail is tool time spent waiting on sub-agents, which ran under the same load. Without that wait, the median is 9 minutes and p90 26.
The median is per review, so it is not the product of the median step count and the median step time.
47 hit it in their first 10 minutes, on a workspace that was still running from an earlier review. 112 hit it partway through, typically about 100 minutes in.
The 401s follow from slow reviews, because a review past an hour outlives its token, so they don't explain the step times.
Why stop instead of report: #31707 (closed) kept the 40-minute wait and turned a timeout on a running chat green. It assumed the review would still post later, but most of those reviews end on the 401, and the ones that keep running are the load.
Why interrupt sub-agents one by one: I found no code in
chatdthat cascades an interrupt from a parent chat to its children.Why
if: always()on the last step: GitHub runsalways()steps when a run is cancelled, so a run that a newer push cancels still stops its review before the newer run starts.The concurrency group is on the job, so a run whose job is skipped (for example, a label other than
doc-check) doesn't cancel the running review.The script starts no poll after its 270-second budget, inside the 5 minutes GitHub gives an
always()step after a cancel. A sweep can run past the budget on a slow deployment, and the steps' 6-minutetimeout-minutesstops a script that hangs.Why chats before workspaces: a review that is still running when its workspace stops gets tool errors, and chatd's system prompt tells a root chat to call
start_workspaceon a stopped workspace (coderd/x/chatd/prompt.go:58). So the chat phase interrupts and waits first, and only then does the workspace phase stop the workspaces.Tradeoffs Nick accepted: a new push discards the review of the older commit, and a review past 25 minutes is stopped and fails the check.
By the unloaded estimate above, that stops 48 of the 134 reviews (36%), mostly ones waiting on sub-agents. Without those waits, it would stop 16. The next PR's single-agent default targets those reviews, rather than a longer wait that keeps them loading the model.
A push also cancels a manual run's review. On a PR whose changed files all skip review, the push run then skips too, so the workflow comment says to dispatch again after the push or add the label.
Next (separate PR): fewer steps per review. The workflow builds the review packet (PR metadata, diff, comments, changed docs) and runs Vale before the chat starts, and the skill uses one agent by default, with sub-agents only for large docs diffs.
Review findings not taken (accepted by the reviewer):
PR_URL: kept, because parsers here tolerate CRLF input by policy, and it costs one line.doc-check-pr-<N>: the chat createddoc-check-pr-31720in the end-to-end test, and a review whose workspace is stopped can start it itself.Follow-ups found in review (not changed here):
coder-workflow-bottemplate mints its GitHub installation token once at startup, so any session past an hour gets 401s.owner/repo, but the workspace name uses only the PR number. The stop script now skips the name lookup for other repositories; the other steps still use the number alone.Generated by Coder Agents on behalf of @nickvigilante.