Skip to content

ci(.github): stop each doc-check review when its run ends - #31720

Open
nickvigilante wants to merge 9 commits into
mainfrom
vigilante/docs-1478-doc-check-stop-stale-reviews-and-expired-token-reruns-from
Open

nickvigilante wants to merge 9 commits into
mainfrom
vigilante/docs-1478-doc-check-stop-stale-reviews-and-expired-token-reruns-from

Conversation

@nickvigilante

@nickvigilante nickvigilante commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

doc-check reviews on the local Solstice model take a median of 150 minutes, so most runs time out.
The skill isn't the bottleneck: at an unloaded step time, the same reviews would take a median of 13 minutes.
The model is slow because doc-check alone keeps a median of 40 review chats active, and most of that work is wasted:

  • A review keeps running after its job stops waiting, and the next push queues another full review behind it.
  • A workspace left running past an hour holds an expired GitHub token, so the review that runs in it hits a 401 and posts nothing.

This PR makes each run stop its review when the run ends, so active reviews don't outnumber running jobs unless a stop fails, and every review starts with a fresh token. The stop steps warn when a stop fails.

Linear: DOCS-1478

Changes

  • cancel-in-progress: true: a new push cancels the PR's run in progress instead of queueing behind it.

  • New .github/scripts/doc-check-stop-review.sh runs two phases, each with one sweep function per kind of item:

    • Chats (the PR's review chats, found by the chat action's labels): a sweep drops the queue of every busy chat in the tree, because finishing an interruption promotes the next queued message, then interrupts each running chat and sub-agent.
      A failed interrupt or queue delete is tried again after a new read, because chatd answers 409 or 404 when a turn ends between the read and the request.
      It also drops the queue of a chat in error, because the next message sent to that chat runs the oldest queued one first.
      A queue failure warns once per chat, because GitHub keeps only the first 10 warning annotations of a step, and a chat that is still running after its last interrupt failed gets its own warning.
    • Workspaces (the chats' workspaces and doc-check-pr-<N>): a sweep waits out a build in progress and sends a stop to any other workspace that isn't stopped, including one whose earlier stop failed.
      A failed stop request is sent again on the next sweep, because coderd answers 409 when a build started after the read.
      A stop build after a failed request counts as this run's stop, because a request that timed out may have been accepted.

    Each phase sweeps every item once, then sweeps the ones that haven't settled until its deadline: the chats get the first third of STOP_BUDGET_SECONDS (270 seconds by default), and the workspaces the rest.
    Each request gets the time left in its phase, at least 1 second and at most 30. The deadline decides whether a poll starts, not when its sweep ends, so a slow deployment can push the script past the budget.
    It looks up doc-check-pr-<N> only for this repository's PRs, because the name has only the number.

  • The job runs the script before the review (Stop the previous review and its workspace) and after every run, including a cancelled one (Stop the review and its workspace, if: always()).
    The existing step that starts a stopped workspace then brings it up fresh, so the startup script mints a new installation token.

  • The concurrency group is keyed on the PR URL, so any new run for a PR, including a manual one, cancels the run in progress instead of stopping its review mid-run.

  • A PR with the doc-check label gets a review on every push until the label is removed, because a push now cancels the labeled run in progress. The skip summary and the file header say so.

  • The wait drops from 40 to 25 minutes, and timeout-minutes from 50 to 45. A review past the wait is stopped, the job fails, and the step summary says the job tried to stop it.

  • The old Stop the review workspace step is gone. It needed the action's workspace-id output, which the action doesn't set on a timeout, so it never ran for the reviews that most needed it.

Testing

  • End to end on dogfood, with workflow_dispatch runs of this branch against this PR. At f038096, the head before 41f8f18:

    • Fresh start (run 1): Prepare the review workspace started the stopped workspace (04:39:48 to 04:39:59 UTC).
    • Cancel (run 1): run 2, dispatched at 04:40:59, cancelled it at 04:42:13. Its last step interrupted the chat, which was waiting 5 seconds later, and stopped doc-check-pr-31720 (04:42:19 to 04:42:51).
    • Queueing (run 2): the job stayed pending until run 1's last step finished at 04:42:51, and started at 04:42:55. Stop the previous review and its workspace found nothing to stop.
    • Manual cancel (run 2): gh run cancel at 04:43:57 stopped the chat and the workspace the same way (04:44:10 to 04:44:26).

    At bf1aa8b, an earlier head, runs 41742809326 and 41743065015 showed the same cancel and queueing results, plus:

    • Fresh token: on the started workspace, the review's gh pr diff and gh pr view calls returned the PR.
    • Timeout: the wait ended at 1500 seconds (00:23:14) and the job failed. The last step interrupted the chat, which was waiting 5 seconds later, and stopped the workspace (00:23:21 to 00:23:32).
      A timeout runs the same script call as a cancel, and the later commits change only the script and workflow comments, so the 25-minute timeout run wasn't repeated.

    The timeout run's review made 6 model steps in 25 minutes, at a median of 339 seconds a step, because 20 other doc-check reviews from before this PR were still running on the model (40 active chats counting sub-agents, at 23:59).

    Not covered end to end:

    • A manual run overlapping a pull request run. The job skips pull request events on a draft PR, so this PR can't produce one. Both runs key the group on the same URL: the PR's html_url is https://github.com/coder/coder/pull/31720, the form pr_url is validated to.
    • Interrupting sub-agents. No review spawned one, so only the script test covers it.
    • Slow or failed requests. Only the script test covers them.
    • 41f8f18's failure paths and the queue of a chat in error. Only the script test covers them.
      The timeout path at 41f8f18 ran on this PR's ready_for_review run (42681464271): the wait ended at 1500 seconds (19:39:51 UTC) and the job failed. The last step interrupted the chat, which the next poll read as stopped 5 seconds later, and stopped doc-check-pr-31720 (19:39:52 to 19:40:09).
  • bash .github/scripts/doc-check-stop-review.test.sh runs the real script against a fake Coder API: a fake curl, sleep, and date on PATH, sharing a virtual clock that slow fake requests also advance. No case waits in real time, and a case fails if it polls past 3000 virtual seconds. It has 48 cases.
    They cover running and idle reviews with sub-agents and queued messages, two matching reviews, a first review whose slow queue read uses up the chat phase, a full queue of 20 and one of 21, messages queued during an interruption, a message chatd promotes between the queue read and the delete, requires_action and paused chats, a chat in error with queued messages, a chat that ignores interrupts, two workspaces where one build never ends, each build status in progress (the fake answers a stop with 409 during one), a failed start, a stop build that failed in this run or an earlier one, a stop request that times out after coderd accepts it, a failed stop request on a workspace another build stopped, a deleted workspace, the poll rule and the request limits (30-second cap, time left in the phase, 1-second floor), the exact workspace name match, another repository's PR, a mixed-case repository URL, and a PR URL with a trailing CR.
    For each of the 8 requests the script makes, at least one case fails it: the chat and workspace lookups, the chat and workspace reads (failing always, then recovering, and after a success), the queue read and delete, the interrupt (failing once and always), and the stop (failing always, once, before a build starts, and after coderd accepts it).
    At f038096, I ran the first 88 one-line breaks in the table below against the suite, and each failed at least one case.
    Two of them failed by crashing on an unset variable, which set -u would also do in production.
    At 41f8f18, I ran the last 6, one for each behavior fix in that commit, and each failed its new case.
    The breaks don't cover every possible edit.
    At 41f8f18, on this 128-core workspace, 8 copies of the suite at once passed in 18 seconds, 4 copies pinned to 2 cores (taskset -c 0,1) in 25, and 8 copies pinned to 1 core in 108.
    No CI job runs shell tests, so this runs by hand on Linux, like the existing .github/workflows/test-*.sh scripts. The script and the test need bash 4.4 or later, which ubuntu-latest has.

  • At 41f8f18, shfmt, shellcheck, make lint/actions/actionlint, and make lint/actions/zizmor pass, with no zizmor findings.

Mutation check: 94 one-line breaks, all caught
Function Break Result First case that failed
api drop the 30-second request cap killed running-review
api drop the 1-second request floor killed chat-budget
api drop --max-time killed running-review
sleep_if_time poll while any time is left killed build-never-ends
sleep_if_time > to >= killed build-never-ends
sleep_if_time always poll killed stuck-chat
chat_tree read only the review chat, not its sub-agents killed running-review
clear_queue return at once killed running-review
clear_queue one pass only killed running-review
clear_queue <= to < in the loop bound killed full-queue
clear_queue drop the failed-read warning killed queue-read-fails
clear_queue retry a failed read in the loop killed queue-read-fails
clear_queue break instead of return on an empty queue killed running-review
clear_queue never warn about a failed delete killed queue-delete-fails
clear_queue warn after any failed delete, even if the first message changed killed queue-promoted
clear_queue drop the failed-delete warning killed queue-delete-fails
clear_queue keep deleting after that warning killed queue-delete-fails
clear_queue delete on the last, read-only pass killed queue-overflow
clear_queue forget which delete failed killed queue-delete-fails
clear_queue return on a failed delete killed queue-promoted
clear_queue drop the 20-deletes warning killed queue-overflow
sweep_chat settle a chat whose read failed killed chat-read-fails
sweep_chat never record a successful read killed stuck-chat
sweep_chat remove running from the busy statuses killed running-review
sweep_chat remove requires_action from the busy statuses killed requires-action
sweep_chat remove interrupting from the busy statuses killed running-review
sweep_chat remove paused from the busy statuses killed paused-chat
sweep_chat never mark the tree busy killed running-review
sweep_chat skip clear_queue killed running-review
sweep_chat invert the idle check killed idle-review
sweep_chat print "are stopped" on the first sweep killed idle-review
sweep_chat never print "are stopped" killed running-review
sweep_chat don't interrupt requires_action killed requires-action
sweep_chat also interrupt interrupting killed running-review
sweep_chat cap interrupts at 1 killed requires-action
sweep_chat no interrupt cap killed stuck-chat
sweep_chat don't count interrupts killed stuck-chat
sweep_chat drop the failed-interrupt line killed interrupt-fails
sweep_chat warn on a failed interrupt killed interrupt-fails
sweep_chat settle a chat once interrupted killed running-review
report_chat always say "did not stop in time" killed chat-read-fails
report_chat always say "Could not read" killed stuck-chat
sweep_workspace settle a workspace whose read failed killed workspace-read-fails
sweep_workspace never record a successful read killed build-never-ends
sweep_workspace keep an earlier successful read after a failure killed workspace-read-lost
sweep_workspace drop include_deleted=true killed build-deleting
sweep_workspace don't wait on pending killed build-pending
sweep_workspace don't wait on starting killed build-starting
sweep_workspace don't wait on stopping killed running-review
sweep_workspace don't wait on canceling killed build-canceling
sweep_workspace don't wait on deleting killed build-deleting
sweep_workspace keep a failed stop once a build starts killed stop-fails-then-build
sweep_workspace forget a sent stop once a build starts killed running-review
sweep_workspace stop a stopped workspace killed idle-review
sweep_workspace stop a deleted workspace killed build-deleting
sweep_workspace print "Stopped workspace" for one this run didn't stop killed idle-review
sweep_workspace never print "Stopped workspace" killed running-review
sweep_workspace never take the after-stop branch killed failed-stop
sweep_workspace always take the after-stop branch killed running-review
sweep_workspace always print the start-build warning killed failed-stop
sweep_workspace never print the start-build warning killed running-after-stop
sweep_workspace stop again after the after-stop warning killed failed-stop
sweep_workspace record a successful stop as failed killed running-review
sweep_workspace don't record a failed stop killed stop-fails
sweep_workspace invert the stop result killed running-review
sweep_workspace settle a workspace once the stop is sent killed running-review
report_workspace never say "Could not read" killed workspace-read-fails
report_workspace always say "Could not read" killed build-never-ends
report_workspace never say "Could not stop" killed stop-fails
report_workspace always say "Could not stop" killed build-never-ends
run_phase no polls killed running-review
run_phase poll with nothing pending killed idle-review
run_phase don't reset the unsettled list killed running-review
run_phase keep settled items in later polls killed running-review
run_phase keep every item after the first sweep killed idle-review
run_phase skip the report killed stuck-chat
main give the chat phase the whole budget killed chat-budget
main don't reset the deadline for the workspace phase killed chat-budget
main stop only the first matching chat killed two-reviews
main look up doc-check-pr-<N> for any repository killed other-repository
main compare repositories case-sensitively killed mixed-case-repository
main match the workspace name as a substring killed exact-workspace-name
main keep a trailing CR on PR_URL killed crlf-pr-url
main skip the chats' own workspaces killed two-workspaces
main drop the chat lookup warning killed budget-caps-requests
main drop the workspace lookup warning killed workspace-lookup-fails
main accept only /pull/ URLs killed running-review
main don't dedupe workspace IDs killed running-review
clear_queue warn on every failed queue request, not once per chat killed queue-read-fails
sweep_chat skip clear_queue for a chat in error killed error-chats
sweep_chat never record a failed interrupt killed interrupt-never-succeeds
sweep_workspace print "Stopped workspace" after a failed stop request killed stop-fails-then-stopped
sweep_workspace clear a failed stop request when a stop build starts killed stop-times-out-after-apply
main keep a null workspace_id killed two-reviews
Unrelated CI flake

At af47edc, test-go-pg (macos-latest) failed in cli TestUpdateTemplateVersionApplyFailure/true (workspace_template_version_test.go:319: expected "partial start state", got ""). At 10eb134, Storybook failed in oauth2appssettingspageview--update-error-outranks-stale-load-error (play function could not find its error text). This PR changes no Go or frontend code. I re-ran each failed job, and both passed.

Decision log

Measurements: every doc-check-bot chat shared with the organization since Oct 30 17:54 UTC, read from GET /api/v2/chats and each chat's messages: 244 parent review turns and 468 sub-agent turns, each timed from its user message to its last message.

  • Finished reviews: median 150 minutes, p90 655. Only 12 of the 159 finished turns with at least 5 model steps took 20 minutes or less. 83% of review time is waiting on the model.
  • A median review makes 29 model steps. With up to 31 doc-check chats active, the median step took 11 to 22 seconds. With 32 to 47 active, it took 221 seconds.
  • The unloaded estimate behind the 25-minute wait: for each of the 134 reviews with at least 5 model steps in chats that were idle when measured, its model steps × 15 seconds (the middle of 11 to 22) plus its measured tool time. The median is 13 minutes, p75 40, and p90 90, and 86 of the 134 fit in 25 minutes.
    Most of the tail is tool time spent waiting on sub-agents, which ran under the same load. Without that wait, the median is 9 minutes and p90 26.
    The median is per review, so it is not the product of the median step count and the median step time.
  • doc-check keeps a median of 40 chats active (p90 66, max 81). At one point on Nov 3, 24 parent chats and 7 sub-agents were running, with 17 more reviews queued behind busy chats.
  • Separately, 159 of all 244 turns hit a GitHub 401. It is a coincidence that this matches the 159 above.
    47 hit it in their first 10 minutes, on a workspace that was still running from an earlier review. 112 hit it partway through, typically about 100 minutes in.
    The 401s follow from slow reviews, because a review past an hour outlives its token, so they don't explain the step times.

Why stop instead of report: #31707 (closed) kept the 40-minute wait and turned a timeout on a running chat green. It assumed the review would still post later, but most of those reviews end on the 401, and the ones that keep running are the load.

Why interrupt sub-agents one by one: I found no code in chatd that cascades an interrupt from a parent chat to its children.

Why if: always() on the last step: GitHub runs always() steps when a run is cancelled, so a run that a newer push cancels still stops its review before the newer run starts.
The concurrency group is on the job, so a run whose job is skipped (for example, a label other than doc-check) doesn't cancel the running review.
The script starts no poll after its 270-second budget, inside the 5 minutes GitHub gives an always() step after a cancel. A sweep can run past the budget on a slow deployment, and the steps' 6-minute timeout-minutes stops a script that hangs.

Why chats before workspaces: a review that is still running when its workspace stops gets tool errors, and chatd's system prompt tells a root chat to call start_workspace on a stopped workspace (coderd/x/chatd/prompt.go:58). So the chat phase interrupts and waits first, and only then does the workspace phase stop the workspaces.

Tradeoffs Nick accepted: a new push discards the review of the older commit, and a review past 25 minutes is stopped and fails the check.
By the unloaded estimate above, that stops 48 of the 134 reviews (36%), mostly ones waiting on sub-agents. Without those waits, it would stop 16. The next PR's single-agent default targets those reviews, rather than a longer wait that keeps them loading the model.
A push also cancels a manual run's review. On a PR whose changed files all skip review, the push run then skips too, so the workflow comment says to dispatch again after the push or add the label.

Next (separate PR): fewer steps per review. The workflow builds the review packet (PR metadata, diff, comments, changed docs) and runs Vale before the chat starts, and the skill uses one agent by default, with sub-agents only for large docs diffs.

Review findings not taken (accepted by the reviewer):

  • Remove the CR strip on PR_URL: kept, because parsers here tolerate CRLF input by policy, and it costs one line.
  • Share one exact-name workspace lookup across the three steps that use it: the other two steps predate this PR, so that refactor is separate.
  • Cap how many PRs review at once: tracked in DOCS-1480, after this PR's effect is measured, because the cap's threshold depends on it.
  • Start the stopped workspaces by ID, in case a chat named its workspace something other than doc-check-pr-<N>: the chat created doc-check-pr-31720 in the end-to-end test, and a review whose workspace is stopped can start it itself.

Follow-ups found in review (not changed here):

  • No CI job runs the repository's shell test scripts.
  • The coder-workflow-bot template mints its GitHub installation token once at startup, so any session past an hour gets 401s.
  • Prepare the review workspace says "Workspace state is usable" after a failed build.
  • The dispatch URL check accepts any owner/repo, but the workspace name uses only the PR number. The stop script now skips the name lookup for other repositories; the other steps still use the number alone.
  • chatd's only concurrent-agent cap is deployment-wide, so a deployment can't cap one model or one bot user.
  • Reviews still running from before this PR keep running until their PR gets another run. Running the stop script once for each open PR with a doc-check chat after merge would stop them.

Generated by Coder Agents on behalf of @nickvigilante.

doc-check reviews on the local model took a median of 150 minutes,
because doc-check alone kept about 40 review chats active and each model
step slowed from about 20 seconds to over 200. Most of that load was
wasted: a review kept running after its job stopped waiting, the next
push queued another full review behind it, and a workspace left running
past an hour held an expired GitHub token, so 159 of 244 reviews ended
on a 401.

A new push now cancels the PR's run in progress. A new script stops the
PR's review chat and its sub-agents, drops their queued messages, and
stops the review workspace. The job runs it before the review, so the
review starts on an idle chat and a freshly started workspace, and after
every run, including a cancelled one. The wait drops to 25 minutes, and
a review past it is stopped and fails the job.

Refs DOCS-1478
@linear-code

linear-code Bot commented Nov 3, 2026

Copy link
Copy Markdown

DOCS-1478

@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-11-05 18:07 UTC by @nickvigilante
Spend: $132.41 / $250.00

Review history
  • R1 (2026-11-03): 12 reviewers, 6 Nit, 2 P2, 7 P3, 13 P4, COMMENT. Review
  • R2 (2026-11-03), 6 Nit, 2 P2, 7 P3, 13 P4, COMMENT. Review
  • R3 (2026-11-03): 10 reviewers, 9 Nit, 2 P2, 10 P3, 22 P4, COMMENT. Review
  • R4 (2026-11-03), 9 Nit, 2 P2, 10 P3, 22 P4, COMMENT. Review
  • R5 (2026-11-04): 11 reviewers, 12 Nit, 2 P2, 12 P3, 33 P4, COMMENT. Review
  • R6 (2026-11-04): 11 reviewers, 15 Nit, 2 P2, 13 P3, 43 P4, COMMENT. Review
  • R7 (2026-11-04): 11 reviewers, 17 Nit, 3 P2, 14 P3, 55 P4, COMMENT. Review
  • R8 (2026-11-04): 13 reviewers, 21 Nit, 4 P2, 16 P3, 65 P4, COMMENT. Review
  • R9 (2026-11-05): 10 reviewers, 25 Nit, 4 P2, 17 P3, 70 P4, COMMENT. Review

deep-review v0.13.0 | Round 9 | bc5dd20..f038096

Last posted: Round 9, 116 findings (4 P2, 17 P3, 70 P4, 25 Nit), COMMENT. Review

Finding inventory

Finding inventory: PR #31720

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P3 Author fixed (5a270bc) doc-check.yaml:62 cancel-in-progress cancels a label-forced review; the push run may skip review on nodocs PRs R1 Netero Yes
CRF-2 P4 Author fixed (5a270bc) doc-check-stop-review.sh:175 Failed workspace lookup prints no warning R1 Netero Yes
CRF-3 P4 Author fixed (5a270bc) doc-check-stop-review.sh:53 clear_queue swallows read and delete failures silently R1 Netero Yes
CRF-4 P4 Author contested; panel closed R3 (1/1 accept) doc-check-stop-review.sh:147 CR strip handles input that cannot occur R1 Netero Yes
CRF-5 P4 Author fixed (5a270bc) doc-check-stop-review.test.sh:246 Four warning branches have no test case R1 Netero Yes
CRF-6 P4 Author contested; panel closed R3 (1/1 accept, refactor moved to out of scope CRF-49) doc-check-stop-review.sh:174 Exact-name workspace lookup duplicated in three places R1 Netero Yes
CRF-7 Nit Author fixed (5a270bc) doc-check.yaml:153 ci_docs omits doc-check-stop-review.test.sh R1 Netero Yes
CRF-8 Note Note (body) doc-check.yaml test-go-pg CI failures are unrelated to this diff R1 Netero No
CRF-9 P2 Author fixed (5a270bc) doc-check.yaml:61 Dispatch and pull_request runs for one PR are in different groups and now stop each other's review; the interrupted run goes green with no review R1 Mafuuu P2, Takumi P3, Hisoka P3, Mafu-san P3, Meruem P3, Leorio P3 Yes
CRF-10 P2 Author fixed (PR description: runs 41742809326, 41743065015 at bf1aa8b) doc-check.yaml:56 Cancel ordering, deployed interrupt behavior, and fresh-token claim never run end to end R1 Mafu-san P2, Leorio P3 Yes
CRF-11 P3 Author fixed (5a270bc) doc-check-stop-review.sh:124 stop_workspace reports a failed or canceled stop build as "Stopped" R1 Meruem, Leorio, Chopper Yes
CRF-12 P3 Author contested; panel closed R3 (3/3 accept; stale-tree half fixed in 5a270bc) doc-check-stop-review.sh:62 paused omitted from any_active and stale tree checked after clear_queue; a restarted chat is reported stopped R1 Gon P3, Melody P4, Bisky P4 Yes
CRF-13 P3 Author fixed (5a270bc) doc-check-stop-review.sh:37 curl has no --max-time; one hung request blocks cleanup until the job or cancel limit R1 Meruem Yes
CRF-14 P3 Deferred (DOCS-1480) doc-check.yaml:490 No bound on concurrent reviews across PRs; a burst past the model's knee times out every review R1 Pariston Yes
CRF-15 P3 Author fixed (5a270bc) doc-check-stop-review.test.sh:110 Fake API finishes transitions on the next read, so no case proves the script waits R1 Bisky Yes
CRF-16 P3 Author fixed (PR description: sets are distinct) doc-check.yaml:490 Two different measurement sets share the count 159 without saying how they relate R1 Mafu-san Yes
CRF-17 P4 Author fixed (5a270bc) doc-check-stop-review.sh:96 requires_action chat back in running waits a full 60 s round before the second interrupt R1 Takumi Yes
CRF-18 P4 Author contested; panel closed R3 (2/2 accept) doc-check-stop-review.sh:178 Script stops workspaces found by chat workspace_id, but Prepare restarts only the one found by name R1 Meruem P4, Melody Note Yes
CRF-19 P4 Author fixed (5a270bc) doc-check-stop-review.test.sh:262 archived-chat case and archived filters cover a state the list endpoint does not return R1 Bisky Yes
CRF-20 P4 Author fixed (5a270bc) doc-check.yaml:490 A review stopped at the wait fails the check with no step summary R1 Chopper Yes
CRF-21 P4 Author fixed (5a270bc) doc-check-stop-review.sh:142 Build-timeout warning names the workspace by ID, other lines by name R1 Chopper P4, Gon Nit, Leorio Nit Yes
CRF-22 P4 Author fixed (5a270bc) doc-check-stop-review.sh:6 Script header repeats the callers' rationale and describes history R1 Gon Yes
CRF-23 P4 Author fixed (5a270bc) doc-check-stop-review.sh:70 Interrupt round bound is an unnamed literal in two places R1 Gon Yes
CRF-24 P4 Author fixed (5a270bc) doc-check-stop-review.sh:53 limit=1 reads as a queue limit; the full queue comes back on any uncursored read R1 Gon Yes
CRF-25 Nit Author fixed (5a270bc) doc-check-stop-review.sh:60 Comments say "generating" for a predicate that also matches requires_action and interrupting R1 Gon Yes
CRF-26 Nit Author fixed (5a270bc) doc-check-stop-review.test.sh:2 Test header restates setup and omits that no CI job runs it R1 Gon Yes
CRF-27 Nit Author fixed (5a270bc) doc-check.yaml:268 Two steps run the same script under different names; checkout step uses a third R1 Gon Yes
CRF-28 Nit Author fixed (5a270bc) doc-check-stop-review.test.sh:131 run_case calls its jq-expression argument STATE_JSON R1 Gon Yes
CRF-29 Nit Author fixed (5a270bc) doc-check.yaml:489 Comment says about 15 minutes; description says median 13 R1 Mafu-san Yes
CRF-30 Note Note (body) doc-check.yaml:499 Cleanup worst case (about 360 s) exceeds GitHub's 5-minute limit for always() steps after a cancel R1 Takumi No
CRF-31 Note Note (body) doc-check.yaml:256 Job now runs a checked-out script with the session token; no new capability for same-repo PRs R1 Kurapika No
CRF-32 Note Dropped by orchestrator (confirmation, asks no change) doc-check-stop-review.sh:160 Chat lookup scoped to bot-owned chats R1 Kurapika No
CRF-33 Note Dropped by orchestrator (confirmation, asks no change) doc-check-stop-review.test.sh:184 Test claims and five breaks reproduced R1 Mafu-san No
CRF-34 Note Dropped by orchestrator (confirmation, asks no change) doc-check-stop-review.sh:65 chatd statements in script match repo code R1 Mafu-san No
CRF-35 P2 Dropped by orchestrator (comment states what each constant bounds, which the code does not show) doc-check-stop-review.sh:24 Polling comment multiplies constants R1 Gon No
CRF-36 P2 Dropped by orchestrator (sentence names both sources at once; no consequence) doc-check-stop-review.sh:172 First comment sentence restates jq lines R1 Gon No
CRF-37 P2 Dropped by orchestrator (one redundant clause in an otherwise rationale comment) doc-check.yaml:497 Comment restates step name R1 Gon No
CRF-38 P2 Dropped by orchestrator (section divider in a long test file; no consequence) doc-check-stop-review.test.sh:182 "--- Cases ---" carries no information R1 Gon No
CRF-39 P2 Dropped by orchestrator (subsumed by CRF-4, which removes the case and comment) doc-check-stop-review.test.sh:281 CR comment repeats script and case name R1 Gon No
CRF-40 P4 Dropped by orchestrator (22 is curl's documented --fail code inside a fake curl) doc-check-stop-review.test.sh:51 exit 22 unexplained R1 Gon No
CRF-41 OOS Out of scope (body) doc-check.yaml:215 Dispatch URL check accepts any owner/repo but workspace name uses only the PR number R1 Kurapika No
CRF-42 OOS Out of scope (body) .github/workflows/test-*.sh No CI job runs the repo's shell test scripts R1 Bisky No
CRF-43 OOS Out of scope (body) coderd/x/chatd/agentadmission.go:12 chatd agent cap is deployment-wide and fixed, not per model or user R1 Pariston No
CRF-44 OOS Out of scope (body) coder-workflow-bot template Workspace mints its GitHub token once at startup R1 Pariston No
CRF-45 OOS Out of scope (body) doc-check.yaml:352 Workspace start wait duplicates the 180-second budget inline R1 Gon No
CRF-46 OOS Out of scope (body) doc-check.yaml:342 Prepare step prints "usable" for a failed stop or start build R1 Meruem No
CRF-47 P4 Dropped by orchestrator (GitHub docs: concurrency group names are case insensitive, so a mixed-case pr_url shares the group) doc-check.yaml:63 Mixed-case dispatch URL puts the run in a separate group R3 Netero No
CRF-48 P4 Author fixed (bf1aa8b) doc-check.yaml:520 Step summary hard-codes "25 minutes", repeating wait-timeout-seconds R3 Netero Yes
CRF-49 OOS Out of scope (body) doc-check.yaml:304 Exact-name doc-check-pr-N lookup in three places (CRF-6 refactor, no ticket) R3 Netero No
CRF-50 P3 Author fixed (bf1aa8b) doc-check-stop-review.sh:18 Comment claims a 5-minute bound; poll counts exclude request time and multiply per chat and workspace R3 Gon P3, Meruem P4, Leorio P4, Mafu-san P4 Yes
CRF-51 P3 Author fixed (bf1aa8b) doc-check-stop-review.sh:147 stop/failed or stop/canceled left by an earlier run warns but is never retried R3 Chopper P3, Meruem P4 Yes
CRF-52 P4 Author fixed (bf1aa8b) doc-check.yaml:520 Timeout summary says the job stopped the review even when the script warned it did not R3 Leorio, Mafuuu Yes
CRF-53 P4 Author fixed (bf1aa8b) doc-check.yaml:64 A push cancels a forced dispatch review on a nodocs PR and the push run skips R3 Mafuuu Yes
CRF-54 P4 Author fixed (bf1aa8b) doc-check-stop-review.sh:175 Dispatch for another repo's PR with the same number stops this repo's running review workspace R3 Mafuuu Yes
CRF-55 P4 Author fixed (bf1aa8b) doc-check.yaml:173 Label now forces a review on every push, but user-facing text describes a one-time review R3 Kite Yes
CRF-56 P4 Author fixed (bf1aa8b) doc-check-stop-review.test.sh:78 Fake promotes a paused chat on any delete; real path deletes a promoted row and warns on 404 R3 Bisky Yes
CRF-57 P4 Author fixed (bf1aa8b) doc-check-stop-review.test.sh:244 Interrupt-spacing check aborts the suite under errexit when grep matches nothing R3 Bisky Yes
CRF-58 P4 Author fixed (bf1aa8b) doc-check.yaml:275 timeout-minutes 6 on both cleanup steps has no stated tie to the script's bounds R3 Gon Yes
CRF-59 P4 Author fixed (bf1aa8b) doc-check-stop-review.sh:148 Failed-build warning is not a sentence and omits the consequence R3 Leorio Yes
CRF-60 P3 Author fixed (bf1aa8b) doc-check.yaml:495 13-minute unloaded median has no derivation; decision-log numbers give 5 to 11 minutes R3 Mafu-san Yes
CRF-61 Nit Author fixed (bf1aa8b) doc-check-stop-review.test.sh:243 Test comment "not the end of a wait" describes the round 1 stop_chat R3 Gon (P2, downgraded: no behavior consequence) Yes
CRF-62 Nit Author fixed (bf1aa8b) doc-check-stop-review.sh:121 name="$1" holds the workspace ID until the first read R3 Gon Yes
CRF-63 Nit Author fixed (bf1aa8b) doc-check.yaml:57 Concurrency comment says "new commit", but a labeled run brings no new commit R3 Kite (Note) Yes
CRF-64 Note Dropped by orchestrator (praise and a local mise trust setting, asks no PR change) doc-check-stop-review.test.sh:27 Fake holds waits; test needs MISE_TRUSTED_CONFIG_PATHS locally R3 Kite No
CRF-65 OOS Out of scope (body) doc-check.yaml:50 doc-check label on a merged PR runs a review and can recreate a deleted workspace R3 Kite No
CRF-66 P4 Author fixed (ccecdfa) doc-check-stop-review.sh:150 stop_workspace checks the budget before its first read; a later workspace is never read or stopped and the warning claims a build was observed R5 Netero Yes
CRF-67 P3 Author fixed (ccecdfa) doc-check-stop-review.sh:107 Chat share bounds only the sleeps; slow chat requests use the whole budget and the workspace is never read or stopped R5 Meruem P3, Takumi P4 Yes
CRF-68 P3 Author contested; panel closed R6 (1/1 accept) doc-check.yaml:505 1500-second wait stops reviews the decision log says take 25 to 40 minutes unloaded; a large PR may never get a review R5 Knov Yes
CRF-69 P4 Author fixed (ccecdfa) doc-check-stop-review.sh:14 Header says the budget is the most time the script takes; the last poll runs a request and a sleep past it R5 Gon, Leorio Yes
CRF-70 P4 Author fixed (ccecdfa) doc-check-stop-review.sh:170 Comment on the catch-all branch describes only failed builds; its usual input is start/running R5 Gon Yes
CRF-71 P4 Author fixed (ccecdfa) doc-check-stop-review.sh:173 "is failed again after the stop" is false for a start build begun after this run's stop R5 Leorio Yes
CRF-72 P4 Author fixed (ccecdfa) doc-check-stop-review.sh:140 "did not stop" warning prints when the last poll never read the chat R5 Chopper Yes
CRF-73 P4 Author fixed (PR description Summary qualified) doc-check-stop-review.sh:140 PR Summary states as guaranteed what the script does only when stops succeed R5 Mafu-san Yes
CRF-74 P4 Author fixed (ccecdfa) doc-check-stop-review.sh:47 Neither the 1-second floor nor the 30-second cap on --max-time is tested R5 Bisky Yes
CRF-75 P4 Author fixed (ccecdfa) doc-check-stop-review.sh:219 No case has two matching chats, so "stop every match" is untested R5 Bisky Yes
CRF-76 P4 Author fixed (ccecdfa) doc-check-stop-review.sh:229 No case uses a mixed-case repository URL, so removing the lowercasing passes R5 Bisky Yes
CRF-77 P4 Author fixed (ccecdfa) doc-check-stop-review.sh:153 Workspace read lacks include_deleted, so a deleted workspace returns 410 and warns as a failed read R5 Mafuuu Yes
CRF-78 P4 Author fixed (ccecdfa) doc-check-stop-review.sh:82 clear_queue warns "still has queued messages" after emptying a full 20-message queue R5 Melody Yes
CRF-79 Nit Author fixed (ccecdfa) doc-check-stop-review.sh:30 QUEUE_DELETES caps deletes per clear_queue call, not per chat R5 Gon Yes
CRF-80 Nit Author fixed (ccecdfa) doc-check-stop-review.sh:204 repo (the PR's repository) next to this_repo does not say which is which R5 Gon Yes
CRF-81 Nit Author fixed (ccecdfa) doc-check-stop-review.test.sh:324 expect_calls 54 is 270 / 5 written as a bare number R5 Gon Yes
CRF-82 Nit Dropped by orchestrator (the file has 24 cases; the 25th ^run_case match is the function definition, verified with grep) doc-check-stop-review.test.sh:203 Description says 24 cases R5 Bisky No
CRF-83 P4 Author fixed (af47edc) doc-check-stop-review.sh:130 New entry deadline check in stop_chat: a second review chat reached after a stuck first chat gets no read and no interrupt R6 Knov, Meruem, Bisky Yes
CRF-84 P3 Author fixed (af47edc) doc-check-stop-review.sh:91 Deadline and read-failure branches added in ccecdfa have no case; deleting them passes the suite R6 Mafu-san P3, Chopper P4, Bisky P4, Mafuuu P4 Yes
CRF-85 P4 Author fixed (af47edc) doc-check-stop-review.test.sh:16 Fake sleep drops its argument, so nothing tests that nap cuts a sleep to the time left R6 Bisky Yes
CRF-86 P4 Author fixed (af47edc) doc-check-stop-review.sh:18 Header says no poll starts after its phase ends; poll 0 runs after it in both loops, and the one-second rule is written in three places R6 Gon, Leorio, Mafuuu, Zoro Yes
CRF-87 P4 Author fixed (af47edc) doc-check-stop-review.sh:238 stop_workspace sleeps to the deadline then breaks without reading, so a finished stop is reported unfinished R6 Takumi Yes
CRF-88 P4 Author fixed (af47edc) doc-check-stop-review.sh:92 Last chat poll starts at the deadline: clear_queue warns on an empty queue and requests get 1 second R6 Takumi Yes
CRF-89 P4 Author fixed (af47edc) doc-check-stop-review.sh:92 Queue warnings do not say the queued message restarts the chat after the interrupt R6 Leorio Yes
CRF-90 P4 Author fixed (af47edc) doc-check-stop-review.sh:221 Start-after-stop warning blames an expired token; a start mints a fresh one and the cause is a chat still running R6 Leorio Yes
CRF-91 P4 Author fixed (af47edc) doc-check-stop-review.sh:39 Chat phase end lost its name, the phase switch has no comment, and the two loops size polls from different names R6 Gon P4, Zoro Nit Yes
CRF-92 P4 Author fixed (af47edc) doc-check-stop-review.sh:105 clear_queue loop runs to DELETES_PER_CLEAR inclusive with no comment saying the last pass only reads R6 Gon Yes
CRF-93 Nit Author fixed (af47edc) doc-check-stop-review.sh:56 nap hides that it cuts the sleep to the phase's time left R6 Gon Yes
CRF-94 Nit Author fixed (af47edc) doc-check-stop-review.sh:183 Same failed chat read prints two different warnings R6 Gon Yes
CRF-95 Nit Author fixed (af47edc) doc-check-stop-review.test.sh:134 Fake comment says it answers 410; it answers a generic error R6 Gon Yes
CRF-96 P4 Author fixed (PR description Testing) doc-check.yaml:506 Testing says ccecdfa changes only time limits and warnings; it also adds include_deleted and changes the queue loop R6 Mafu-san (P3, downgraded) Yes
CRF-97 Note Dropped by orchestrator (explains test semantics, asks no change) doc-check-stop-review.test.sh:360 build-never-ends asserts the poll-count backstop, not the deadline R6 Zoro No
CRF-98 P4 Author fixed (10eb134) doc-check-stop-review.sh:135 A chat whose first tree read fails gets its first interrupt with its queue still in place R7 Netero Yes
CRF-99 P2 Author fixed (10eb134) doc-check-stop-review.test.sh:338 chat-budget fails on a loaded machine (about 4 seconds of slack for fake overhead); Testing's 8-copies claim is false R7 Mafu-san P2, Bisky P3, Chopper P3, Gon P3, Knov P4, Mafuuu P4, Leorio P4, Zoro P4 Yes
CRF-100 P3 Author fixed (10eb134) doc-check-stop-review.sh:176 No case has a chat read that changes result between polls; deleting either read_ok update or the line 160 failure branch passes R7 Mafu-san P3, Mafuuu P4, Bisky P4 Yes
CRF-101 P4 Author fixed (10eb134) doc-check-stop-review.sh:145 "the chat phase's deadline limits only the waiting" is false (first-pass requests get max=1) and repeats the header; same text in test.sh:327 and the commit body R7 Gon (P2, downgraded), Leorio P4, Zoro Nit Yes
CRF-102 P4 Author fixed (10eb134) doc-check-stop-review.sh:70 First-pass requests after the chat phase get a 1-second limit, so on the slow coderd that ended the phase the second chat is never interrupted R7 Knov Yes
CRF-103 P4 Author fixed (10eb134) doc-check-stop-review.sh:22 First pass has no deadline, so overrun grows with chats and queues (87 s against a 6 s budget in the fake); "a few seconds past the budget" is false R7 Leorio Yes
CRF-104 P4 Author fixed (10eb134) doc-check-stop-review.sh:306 Workspaces are stopped one at a time; a never-finishing build leaves the next workspace 1-second requests and no wait R7 Takumi Yes
CRF-105 P4 Author fixed (10eb134) doc-check-stop-review.sh:281 Empty roots array aborts under set -u on bash 4.0 to 4.3; PR says bash 4 or later R7 Meruem P4, Mafuuu Note Yes
CRF-106 P4 Author fixed (10eb134) doc-check-stop-review.sh:150 Poll counts can no longer end a loop in production; the comment explaining them was deleted R7 Meruem Yes
CRF-107 P4 Author fixed (10eb134) doc-check-stop-review.sh:135 Queue drop before a repeat interrupt has no case; replacing sent > 0 with 0 passes R7 Bisky Yes
CRF-108 P4 Author fixed (10eb134) doc-check-stop-review.sh:60 Changing sleep_if_time's check to > 0 passes the suite R7 Mafuuu Yes
CRF-109 P4 Author fixed (10eb134) doc-check-stop-review.sh:159 stop_chats repeats the poll body in its first loop instead of running poll 0 without a sleep R7 Zoro Yes
CRF-110 P4 Author fixed (10eb134) doc-check-stop-review.test.sh:64 Fake's scalar slow form has no remaining case R7 Zoro Yes
CRF-111 P4 Author fixed (10eb134) doc-check-stop-review.sh:18 Header guarantee overstates: idle or unreadable chats get no interrupt, a mid-build workspace gets no stop; poll rule repeats sleep_if_time's comment R7 Gon (P2, downgraded), Kite Nit Yes
CRF-112 Nit Author fixed (10eb134) doc-check-stop-review.sh:266 Phase comments in main (lines 266, 283) repeat the header's split R7 Gon (P2, downgraded) Yes
CRF-113 Nit Author fixed (10eb134) doc-check-stop-review.sh:123 interrupt_active and any_active use "active" for different state sets R7 Gon Yes
CRF-114 P4 Author fixed (f038096) doc-check-stop-review.sh:180 No case covers a workspace read that succeeds and then fails; keeping an earlier ws_read_ok true passes R8 Netero, Bisky Yes
CRF-115 P4 Author fixed (f038096) doc-check-stop-review.sh:193 No case fails when "Stopped workspace" prints for a workspace this run did not stop; Testing's 40-mutation claim is false R8 Netero, Bisky Yes
CRF-116 P3 Author fixed (f038096) doc-check-stop-review.test.sh:198 run_case's real-time timeout 60 fails cases on a loaded or small host although the virtual clock made assertions load-independent (two-workspaces 44 to 52 s) R8 Netero P4 (test.sh:481), Komugi P3, Gon Nit Yes
CRF-117 P4 Author fixed (f038096) doc-check-stop-review.sh:136 Non-idle chat status set written twice (regex line 136, case line 144); removing requires_action from line 144 passes R8 Netero, Bisky Yes
CRF-118 P4 Author fixed (f038096) doc-check-stop-review.sh:158 Interrupt that races a chat finishing its turn gets 409 and warns "Could not interrupt chat" for a chat that stopped R8 Takumi, Komugi, Hisoka Yes
CRF-119 P2 Author fixed (f038096) doc-check-stop-review.sh:159 Testing's "every failed request" coverage and "40 one-line breaks each failed" claims are false; at least 10 one-line breaks pass R8 Mafu-san, Bisky Yes
CRF-120 P4 Author fixed (f038096) doc-check-stop-review.sh:99 clear_queue's restart warnings (lines 100, 118) and the return at 101 have no case R8 Bisky Yes
CRF-121 P4 Author fixed (f038096) doc-check-stop-review.sh:213 Failed stop settles the workspace and is never retried; stop-fails passes with the return deleted R8 Meruem, Bisky Yes
CRF-122 P4 Author fixed (f038096) doc-check-stop-review.sh:199 Two ws_stop_sent checks are one condition; the return at 201 has no case R8 Bisky Yes
CRF-123 P3 Author fixed (f038096) doc-check-stop-review.test.sh:423 Fake accepts a stop during a build (coderd answers 409), so build-in-progress passes when the script stops mid-build; deleting */pending passes R8 Bisky Yes
CRF-124 P4 Author fixed (f038096) doc-check-stop-review.sh:65 "a sweep after it fit in the phase" and "the deadline bounds the polling" are false (sh:24, 65, 230, doc-check.yaml:280): a poll can start with 1 second left R8 Gon, Leorio Yes
CRF-125 Nit Author fixed (f038096) doc-check-stop-review.sh:18 Header's second half repeats run_phase, sweep, and api comments R8 Gon (P2, downgraded) Yes
CRF-126 Nit Author fixed (f038096) doc-check-stop-review.sh:124 sweep_chat comment sentence restates the interrupt loop R8 Gon (P2, downgraded) Yes
CRF-127 P2 Dropped by orchestrator (the round 7 review proposed this exact text for CRF-112) doc-check-stop-review.sh:264 # Chat phase. and # Workspace phase. only label the next line R8 Gon No
CRF-128 Nit Author fixed (f038096) doc-check-stop-review.sh:53 interrupts lacks the chat_ prefix and is keyed by every chat in the tree, unlike its neighbors R8 Gon Yes
CRF-129 Nit Author fixed (f038096) doc-check-stop-review.sh:54 Per-item flag maps mix true/false with == and 1 with -n R8 Gon Yes
CRF-130 P4 Author fixed (f038096) doc-check-stop-review.sh:225 "Could not read workspace X." gives no state and shows a UUID when no read succeeded R8 Leorio Yes
CRF-131 P4 Author fixed (f038096) doc-check-stop-review.sh:112 A 404 on a queue delete (message promoted after the read) stops clearing, so the interrupt promotes the next message R8 Komugi Yes
CRF-132 Note Note (body) doc-check.yaml:506 Until reviews started before this PR end, the 25-minute wait stops most new reviews; a human can decide whether to run the stop script for open PRs at merge R8 Pariston No
CRF-133 P4 Open doc-check-stop-review.sh:208 No case tells a failed stop from a sent one at stop/stopped; reverting the == "sent" check to -n passes R9 Netero Yes
CRF-134 Note Note (body) doc-check-stop-review.sh:109 Two more one-line edits pass (report_workspace branch swap, qid == failed check below the last-pass break); each changes only warning text, and Testing says the 88 breaks are not every edit R9 Netero No
CRF-135 P4 Open doc-check-stop-review.sh:203 A stop request that failed or timed out after coderd accepted it ends at stop/stopped with no result line and no warning R9 Takumi, Leorio Yes
CRF-136 P3 Open doc-check-stop-review.sh:102 Queue warnings repeat on every sweep, so a lasting queue failure uses GitHub's 10 warning annotations per step and hides report_chat and report_workspace warnings R9 Leorio Yes
CRF-137 P4 Open doc-check-stop-review.sh:168 Every failed interrupt is silent, so a chat whose interrupts all failed ends with "did not stop in time"; the third "next sweep checks it again" is false R9 Hisoka, Leorio Yes
CRF-138 P4 Open doc-check-stop-review.sh:144 A chat in error keeps its queue, and the next prompt to it promotes the stale queued message first (sendMessageE1) R9 Kite Yes
CRF-139 P4 Open doc-check-stop-review.test.sh:179 Fake answers any workspace ID, so removing // empty (GET and stop on /workspaces/null) passes every case R9 Bisky Yes
CRF-140 Nit Open doc-check-stop-review.sh:51 ws_stop comment says failed holds while no build has started; it is cleared only when a read shows a build in progress R9 Leorio Yes
CRF-141 P2 Dropped by orchestrator (the round 8 review proposed this exact comment for CRF-124) doc-check-stop-review.sh:65 sleep_if_time comment restates its body R9 Gon No
CRF-142 Nit Open doc-check-stop-review.test.sh:53 Fake header says a stop answers 409 during a build; the fake calls fail; current is not listed among workspace fields R9 Gon Yes
CRF-143 Nit Open doc-check-stop-review.test.sh:5 "polls past CLOCK_LIMIT virtual seconds" reads as a duration; CLOCK_LIMIT is an absolute clock reading R9 Gon Yes
CRF-144 Nit Open doc-check-stop-review.test.sh:17 The 3000 in CLOCK_LIMIT has no comment tying it to the budgets it must exceed R9 Gon Yes
CRF-145 Nit Dropped by orchestrator (the round 8 review proposed this exact warning for CRF-130) doc-check-stop-review.sh:238 Workspace read warning differs from the chat read warning R9 Gon No
CRF-146 Note Note (body) doc-check.yaml:68 cancel-in-progress lets a review job entering after a merge (label or dispatch) cancel the delete-workspace job in the same group; no worse than CRF-65 R9 Kite No

Contested and acknowledged

CRF-4 (P4, doc-check-stop-review.sh:147) - CR strip handles input that cannot occur

  • Finding: Netero proposed removing the carriage-return strip on PR_URL, its comment, and the crlf-pr-url case, because PR_URL is built on ubuntu-latest from html_url or the validated dispatch input and cannot end in \r.
  • Author defense: The repository owner's policy is that parsers tolerate CRLF input and test it; the strip costs one line and keeps the script safe if the URL comes from another source.
  • Panel closure (R3, 1/1): Netero, the agent that raised it, accepted: keeping the strip costs one line and one case for input the workflow does not produce, and no behavior breaks.

CRF-6 (P4, doc-check-stop-review.sh:174) - Exact-name workspace lookup duplicated in three places

  • Finding: Netero proposed sharing one exact-name workspace lookup between the script, Prepare the review workspace, and delete-workspace.
  • Author defense: The other two lookups are in steps this PR does not otherwise touch; moving them widens the diff past the load fix. Left for a separate change (no ticket).
  • Panel closure (R3, 1/1): Netero, the agent that raised it, accepted that sharing the lookup means moving two older inline steps into .github/scripts, and moved that refactor to the out-of-scope list for a human decision (CRF-49).

CRF-12 (P3, doc-check-stop-review.sh:62) - paused chat reported stopped

  • Finding: Re-read the tree after clear_queue before any_active, and add paused to any_active.
  • Author defense (partial): stop_chat now re-reads the tree on every poll and interrupts any running chat, with a paused-chat case. paused stays out of any_active, because a paused chat is not generating and cannot be interrupted.
  • Panel closure (R3, 3/3): Gon, Melody, and Bisky, the agents that raised it, verified the re-read on poll 0 and accepted leaving paused out: a paused chat generates nothing, Interrupt has no case for the paused state, and only the bot's token can edit the queued message. Bisky's new finding CRF-56 covers the fake's paused model.

CRF-14 (P3, doc-check.yaml:490) - No bound on concurrent reviews across PRs

  • Finding: Count running doc-check chats before sending the prompt and skip above a threshold.
  • Author defense: Agreed; the threshold depends on post-PR concurrency, so it is measured first. Do not re-evaluate. Tracked in DOCS-1480.

CRF-18 (P4, doc-check-stop-review.sh:178) - Prepare restarts only the workspace found by name

  • Finding: The script stops workspaces found by each chat's workspace_id and by the name doc-check-pr-N, but Prepare the review workspace starts only the named one. Proposed: write stopped IDs to GITHUB_OUTPUT and start those.
  • Author defense: On dogfood the chat created doc-check-bot/doc-check-pr-31720, and the prompt tells a review to call start_workspace on a stopped (not dormant) workspace, so a differently named workspace costs a few model steps, not a failed review.
  • Panel closure (R3, 2/2): Meruem and Melody, the agents that raised it, accepted. chatd tells the model to start a stopped workspace (coderd/x/chatd/prompt.go:58, coderd/x/chatd/chatd.go:114), and that start runs the startup script, so the token is fresh; the cost is one tool error and one start_workspace call.

CRF-68 (P3, doc-check.yaml:505) - 1500-second wait stops long reviews

  • Finding: By the decision log's estimate, 48 of 134 reviews take longer than 25 minutes unloaded, so the wait stops them after 25 minutes of model time; a large PR may be stopped on every push. Proposed: state the count in the accepted tradeoff, or tie the wait to the one-hour token lifetime.
  • Author defense: Kept 1500 seconds and stated the count in the accepted tradeoffs ("stops 48 of the 134 reviews (36%) ... Without those waits, it would stop 16"). The target is a review within about 20 minutes, a longer wait keeps long reviews loading the model, and the next PR moves to one agent by default.
  • Panel closure (R6, 1/1): Knov, the agent that raised it, accepted: the accepted tradeoffs now state the count (48 of 134, 16 without sub-agent waits), which was the first option the finding offered, so the human owner of the tradeoff accepts it with the number visible.

Law analysis

R9 (head f038096): effective LOC reported +1102 -34 (Law: the 704-line test file is counted as production; about 430 production lines). Verdict: Don't split. Enforcement: Advisory. One reviewable idea (each doc-check run stops its own review when it ends); the cancel and label changes are safe only with the post-run stop step.

Round log

Round 1

Netero first pass: 1 P3, 5 P4, 1 Nit, 1 Note. No P0, panel proceeds. Law not run (effective additions 522).
Round type: round 1, full panel. Multi-domain (bash plus GitHub Actions YAML, session-token handling and shell-out), target 9-12. Specialists: Kurapika (secrets, token in CI), Melody (script to Coder API wire format), Takumi (cancellation, always() cleanup, polling). Core: Pariston, Mafuuu. Pool: Bisky (tests), Hisoka, Mafu-san, Gon, Leorio, Chopper. Wildcard (random from distilled set): Meruem.
Panel: 2 P2, 6 P3 (plus Netero P3), 8 P4 (plus Netero 5), 5 Nit (plus Netero 1), 2 Note (body), 9 dropped, 6 out of scope. Reviewed against bc5dd20..38970da.
Cross-check: CRF-9 convergent across six reviewers, set to P2 (Mafuuu, higher wins). CRF-10 kept at P2: keep argument (three unobserved mechanisms decide whether the PR meets its goal) outweighs the draft-status argument. GitHub docs confirm always() steps keep running after a cancel, force-terminated at 5 minutes (CRF-30). Gon's comment findings rated P2 downgraded to P4/Nit or dropped: no behavior consequence.

Round 2 update

BLOCKED. CRF-14 deferred without a ticket ("planned as a follow-up", no issue linked). No review. Churn guard: 24 addressed, 1 acknowledged (CRF-18), 2 contested (CRF-4, CRF-6), 1 deferred without ticket (CRF-14). Status cells unchanged until the panel runs. Head: 5a270bc.

Round 3

Churn guard PROCEED: CRF-14 now tracked in DOCS-1480. Head unchanged (5a270bc). Round type: fix round. Since the last panel round (R1, 38970da) no files were added or renamed, no PR goal changed, effective additions grew 117; stop_chat and stop_workspace were rewritten, but every change traces to a round 1 finding, so this is not a restructure. Netero reused its R1 agent (CRF-4, CRF-6 contested): both closed 1/1; new CRF-48 P4; CRF-47 dropped (GitHub docs: concurrency group names are case insensitive). Panel: dispute check-in reusing R1 agents for Meruem and Melody (CRF-18) and Gon, Melody, Bisky (CRF-12 partial contest). New agents: Takumi (stop_chat poll rewrite, CRF-13/17 fixes), Chopper (CRF-11/20/21 messages and summary), Leorio (mandatory: error messages and comments changed), Mafuuu (CRF-9 concurrency fix), Mafu-san (CRF-10/16 verification claims in PR text). Gon mandatory (comments touched). Wildcard (random): Kite.
Panel: Takumi no findings (CRF-9, CRF-17 verified). Fixes verified by reviewers: CRF-1, 2, 3, 5, 7 (Netero), 9 (Mafuuu, Meruem, Takumi), 11 (Meruem, Chopper, Leorio), 13 (Meruem), 15, 19 (Bisky), 17 (Takumi), 20, 21 (Chopper), 21 to 28 (Gon). CRF-10 re-raised: GitHub API (unauthenticated) returns head_sha 38970da for runs 41729412601 and 41731313388; 5a270bc committed 23:03:21Z after both. New: 3 P3, 9 P4 (plus Netero CRF-48), 3 Nit, 1 dropped, 1 out of scope. CRF-50 convergent at Gon's P3 (higher wins). CRF-51 at Chopper's P3 (reproduced). Gon's CRF-61 P2 downgraded to Nit: keep argument (a stale comment hides which wait the check rules out) loses to no behavior consequence. Reviewed against 38970da..5a270bc.

Round 4 update

BLOCKED. CRF-10 silent: the R3 re-raise (runs tested 38970da) has no author reply, its thread was resolved without a response, and Testing still cites only runs 41729412601 and 41731313388 with no tested SHA. No review. Churn guard: 15 of 16 open findings addressed in bf1aa8b (CRF-48, 50 to 63), status cells unchanged until the panel verifies. Head: bf1aa8b.

Round 5

Churn guard PROCEED: CRF-10 answered (runs 41742809326 and 41743065015, head_sha bf1aa8b confirmed via the public GitHub API; Testing names the SHA and lists uncovered paths). All 16 open findings addressed. Round type: fix round. Since the last panel round (R3, 5a270bc) no files added or renamed, no goal change, effective additions grew 118; stop_chat, stop_workspace, and clear_queue changed under a time budget, every change traces to a round 3 finding. Netero new agent (no contested Netero findings). Panel (new agents): Meruem, Chopper (CRF-50/51), Gon and Leorio (mandatory: comments and messages changed), Bisky (CRF-56/57, tests), Mafuuu (CRF-52/53/54), Mafu-san (CRF-10/60 claims), Kite (CRF-55/63), Takumi (time budget and poll timing), Melody (clear_queue re-read against the queue API). Netero R5: CRF-66 P4. Wildcard (random): Knov.
Panel results: Kite no findings; Takumi P4 merged into CRF-67. Fixes verified by reviewers: CRF-10 (Mafu-san: head_sha bf1aa8b for both runs, draft PR blocks a pull request overlap), CRF-48 (Netero), CRF-50 (Gon, Takumi; residual overrun is CRF-69), CRF-51 (Meruem), CRF-53, 54, 55, 63 (Kite), CRF-56, 57 (Bisky), CRF-58, 61, 62 (Gon), CRF-59 (Leorio), CRF-60 (Mafu-san). CRF-67 convergent with CRF-66: slow chat requests spend the budget, then stop_workspace skips its first read, so the workspace keeps its old token; CRF-67 at Meruem's P3 (higher wins), CRF-66 stays P4 (Netero) and is linked in the comment. CRF-68 kept at P3: keep argument (decision log's own numbers put 48 of 134 unloaded reviews past 25 minutes, and a large PR is stopped on every push) outweighs the decision log's caveat that most of the tail is load-inflated sub-agent waits; the tradeoff is listed as accepted by the human in the description, so the comment asks to state the count there or raise the wait. Mafuuu's 410 claim verified at coderd/workspaces.go:92-96. Reviewed against 5a270bc..bf1aa8b.

Round 6

Churn guard PROCEED: 15 addressed in ccecdfa, CRF-68 acknowledged (author took the first proposed option). Round type: fix round (no files added or renamed, no goal change, +102 effective additions since R5; changes trace to R5 findings). Panel: Knov via its R5 agent (CRF-68 dispute check-in); new agents for Meruem and Takumi (CRF-67 phase deadline), Gon and Leorio (comments and messages changed), Chopper (CRF-72), Mafu-san (CRF-73 description), Bisky (CRF-74 to 76 tests), Mafuuu (CRF-77), Melody (CRF-78 queue loop). Netero R6: no findings, CRF-66 verified. Wildcard (random): Zoro.
Panel results: Melody no findings. Fixes verified: CRF-66 (Netero, Knov), CRF-67 (Meruem, Takumi, Knov), CRF-70, 79, 80, 81 (Gon), CRF-72 code (Chopper), CRF-73 (Mafu-san), CRF-74 to 76 (Bisky), CRF-77 (Mafuuu, Melody), CRF-78 (Melody, Knov). CRF-69 and CRF-71 fixes added new false text (CRF-86, CRF-90; CRF-90's sentence came from the R5 review's own proposal). CRF-83 is a regression from ccecdfa's entry check (three reviewers reproduced it). CRF-84 at Mafu-san's P3 (higher wins; same pattern as CRF-5 and CRF-74 to 76). CRF-96 downgraded P3 to P4: keep argument (the Testing scope sentence hides two API changes from end-to-end checking) loses because Mafuuu and Melody verified both changes against coderd source this round and the fix is a one-clause edit. Reviewed against bf1aa8b..ccecdfa.

Round 7

Churn guard PROCEED: 14 of 14 addressed in af47edc (stop_chat replaced by stop_chats, has_time and nap removed, sleep_if_time added). Round type: fix round: no files added or renamed, no goal change, +19 effective additions since R6; the stop_chats rewrite is the fix CRF-83 proposed. Panel (new agents, no contested findings): Knov, Meruem, Bisky (CRF-83), Mafu-san, Chopper, Mafuuu (CRF-84), Gon, Leorio, Zoro (CRF-86, 89 to 95), Takumi (CRF-87, 88). Netero R7: CRF-98 P4. Wildcard (random): Kite.
Panel results: Fixes verified: CRF-83 (Meruem, Bisky, Knov, Netero), CRF-84 deadline half and read_ok branch (Bisky, Chopper, Netero; read_ok updates gap is CRF-100), CRF-85 (Bisky), CRF-86, 89, 90 (Leorio, Zoro), CRF-87, 88 (Takumi, Netero), CRF-91 (Zoro), CRF-96 (Mafu-san). End-to-end runs 41756249763 and 41756429043 ran af47edc (Mafu-san, Kite; public API). CRF-99 convergent across eight reviewers at Mafu-san's P2 (higher wins); Mafu-san verified 8 of 8 passes with STOP_BUDGET_SECONDS=30 and a 10-second queue read. CRF-100 at Mafu-san's P3 (higher wins; fifth instance of untested branches). CRF-101 Gon P2 downgraded to P4: keep argument (the comment contradicts the header and hides that first-pass requests get 1 second, the failure in CRF-102) loses because the behavior gap is its own finding and the same class of false header text was P4 in CRF-86; Zoro's Nit at sh:144 is the same sentence, merged. CRF-111 Gon P2 downgraded to P4 (Kite's sh:18 Nit merged): keep argument (the header promises a stop to every workspace after the deadline, so an operator may not check one) loses because the script prints "did not finish its build" in that case. CRF-112 Gon P2 downgraded to Nit: keep argument (a repeated split can drift from the header) loses to no behavior consequence. CRF-102 and CRF-103 pull opposite ways (more time per first-pass request vs bounding first-pass overrun); posted separately, each comment notes the other. Reviewed against ccecdfa..af47edc.

Round 8

Churn guard PROCEED: 16 of 16 addressed in 10eb134 (stop_chats and stop_workspace replaced by run_phase, sweep_chat, sweep_workspace; the test fake gains a virtual clock, fail_seq, and queue_on_interrupt). Inventory validation: no non-posted item's assumptions changed (CRF-97's poll-count backstop is gone with polls, so it stays dropped). Round type: restructure round: both stop functions were rewritten rather than patched, though every change traces to round 7 findings; no files added or renamed, no goal change, +83 effective additions since R7 (961 total, Law not run).
Netero R8 (new agent, no contested Netero findings): CRF-98 verified; CRF-114 to CRF-117 P4.
Panel (full, multi-domain bash plus GitHub Actions YAML with session-token handling, target 9-12): specialists Takumi (sweep and poll timing, cancel budget), Melody (script to Coder API and fake fidelity), Kurapika (session token in CI); core Pariston, Mafuuu; pool Bisky (tests, required), Komugi (virtual clock determinism), Gon and Leorio (mandatory: restructure round), Mafu-san (Testing claims), Meruem, Hisoka. 12 plus wildcard. All new agents (no contested findings). Wildcard (random from Chopper, Kite, Zoro, Knov, Razor): Knov.
Panel results: Melody, Kurapika, Mafuuu, Knov no findings. Fixes verified: CRF-98 (Netero), CRF-99 (Bisky, Komugi, Gon, Mafu-san: 8 copies passed at load 134), CRF-100 chat side (Bisky, Mafuuu, Mafu-san; workspace side is CRF-114), CRF-101 (Gon), CRF-104 (Takumi), CRF-105, 106 (Meruem, Mafuuu), CRF-107 (Bisky), CRF-108 (Mafuuu), CRF-112, 113 (Gon). CRF-10 at this head: runs 41765521267 and 41765674879 ran 10eb134 (Mafu-san, public API). Melody: the fake does not model the I1 queue promotion, but modeling it passed all cases (no finding). CRF-116 raised to P3 (Komugi reproduced exit 124 on 1 and 2 cores; higher wins); line moved to test.sh:198 where Komugi and Gon reported. CRF-119 kept at P2: keep argument (the author's verification claim that closed CRF-100 and CRF-108 is false with at least 10 surviving breaks, and this is the third round with a Testing claim that does not match a run) outweighs the downgrade argument that each gap is posted separately, because the claim is what a human approver relies on. CRF-124: Gon's sh:18 P2 split; the false bound merged into CRF-124 at P4 (keep argument for P2: an operator sizing timeout-minutes concludes only the first sweep overruns; loses because the same header sentence says a slow deployment can push the script past the budget, and the overrun is about 1 second per request); the repetition is CRF-125, downgraded to Nit (no behavior consequence). CRF-126 Gon P2 downgraded to Nit, same reason. CRF-127 dropped: keep argument (a label that names the next line adds nothing) loses because round 7's review asked for exactly this text and reversing it is churn with no consequence. Bisky's stop-fails assertion folded into CRF-121 (the fix and the test depend on the same retry decision); Bisky's sh:144 mutation folded into CRF-117. Reviewed against af47edc..10eb134.

Round 9

Churn guard PROCEED: 17 of 17 addressed in f038096 (failed stops retried through ws_stop sent/failed, clear_queue continues after a failed delete, one busy status set, failed-interrupt warning removed, fake answers 409 during builds and has a virtual-clock hang guard, new cases). Inventory validation: CRF-127 stays dropped (the labels are unchanged); CRF-132 is a body note on unchanged yaml. Round type: fix round: no files added or renamed, no goal change, +141 effective additions since R8; each change traces to a round 8 finding. Law triage: effective additions 1102 exceed 1000 and grew 580 since round 1 (522), and Law has never run, so Law runs.
Netero R9 (new agent): CRF-114 to 117 verified; CRF-133 P4; CRF-134 Note (body). Law R9: Don't split, Advisory (see Law analysis).
Panel (fix round): personas with open findings in code this commit touches: Bisky (CRF-119 to 123), Komugi (CRF-116, 131), Takumi and Hisoka (CRF-118), Mafu-san (CRF-119 Testing claims), Meruem (CRF-121), Gon (CRF-124 to 129, mandatory: comments changed), Leorio (CRF-124, 130, mandatory: warnings changed). Specialists for the changed files: Takumi (retry and poll timing), Melody (fake 409 model against coderd). All new agents (no contested findings). Wildcard (random from Mafuuu, Chopper, Kite, Zoro, Knov, Razor): Kite.
Workspace outage: the 12-CPU workspace stopped responding (load average about 97) while nine reviewers ran the full suite at once; only Hisoka had written its file. Restarted the workspace and resumed the same nine agents with message_agent, asking for single-case runs instead of the full suite. No reviewer ran the full suite at this head; Bisky, Komugi, Takumi, Mafu-san, Meruem, Gon, Melody ran targeted cases and mutations.
Panel results: Komugi, Mafu-san, Meruem, Melody no findings. Fixes verified: CRF-114 to 117 (Netero), CRF-116 (Komugi: two-workspaces 10 s on one core), CRF-117, 120, 121, 122, 123 (Bisky, mutations), CRF-118 (Komugi, Takumi, Meruem, Hisoka), CRF-119 (Mafu-san: 88-row table, 6 breaks reproduced; Bisky: 20 breaks reproduced), CRF-121 (Meruem, Takumi), CRF-124, 130 (Leorio), CRF-131 (Komugi, Takumi). CRF-10 at this head: runs 41774541251 and 41774675904 ran f038096 (Mafu-san). CRF-135: Takumi sh:203 and Leorio sh:208 merged (same silent outcome; Komugi reproduced the timeout schedule in a note); CRF-133 stays the missing case at the same branch. CRF-136 kept at P3: Leorio reproduced 18 queue warnings before report_chat's, and the timeout summary tells the reader to look for those warnings. CRF-137: Hisoka and Leorio convergent at P4. CRF-138: keep argument for dropping (Melody R8: an errored chat does not restart by itself) does not apply, because Kite traced the restart to the next send (sendMessageE1, transitions.go:614), which the action makes. CRF-141 dropped: keep argument (the comment restates the body) loses because the round 8 review asked for this text. CRF-145 dropped for the same reason. Reviewed against 10eb134..f038096.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds a script that stops a PR's doc-check chat, sub-agents, and workspace before and after each run, sets cancel-in-progress: true, and cuts the wait to 25 minutes. Findings: 2 P2, 7 P3, 13 P4, 6 Nit.

Out of scope (needs a ticket or explicit acceptance by a human):

  • .github/workflows/doc-check.yaml:215: the dispatch URL check accepts any owner/repo, but the workspace name uses only the PR number, so a dispatch for other/repo/pull/123 starts, and now also stops, the workspace of coder/coder PR 123.
  • .github/workflows/test-*.sh: no CI job runs the repository's shell test scripts, including this PR's, so each one stops matching the code it tests without anyone seeing a failure.
  • coderd/x/chatd/agentadmission.go:12: chatd's only concurrent-agent cap is deployment-wide and fixed (5 root, 10 sub-agent slots, or off), so a deployment cannot cap one model or one bot user.
  • coder-workflow-bot template (not in this repository): the workspace mints its GitHub installation token once at startup, so any session past one hour gets 401s.
  • .github/workflows/doc-check.yaml:352: the workspace start wait repeats the same 180-second budget inline (seq 36, sleep 5), so a change to the script's budget leaves this one unchanged.
  • .github/workflows/doc-check.yaml:342: Prepare the review workspace starts the workspace only for stop/stopped and prints "Workspace state is usable." for a failed stop or start build, so the review runs on a workspace whose last build failed.

Notes:

  • The test-go-pg and test-go-pg-17 failures are not caused by this diff; .github/actions/test-go-pg/action.yaml references none of the changed files.
  • The cleanup step's worst case (3 interrupt rounds of 60 s plus a 180 s workspace wait, about 360 s) is longer than the 5 minutes GitHub gives always() steps after a cancel. The newer run's Stop the previous review repeats the stop, so the result is a force-terminated step in the cancelled run's log.
  • The job now runs a checked-out script with DOC_CHECK_CODER_SESSION_TOKEN. For same-repository pull_request runs the PR author already controls the workflow file at the same ref, so this adds no new capability.

🤖 This review was automatically generated with Coder Agents.

Comment thread .github/workflows/doc-check.yaml Resolved
Comment thread .github/workflows/doc-check.yaml Resolved
Comment thread .github/workflows/doc-check.yaml Resolved
Key the concurrency group on the PR URL, so a manual run and a pull
request run for one PR cancel each other instead of stopping each
other's review. Keep forcing a review while the PR has the doc-check
label, because a push now cancels the labeled run.

In the script, interrupt again on the next poll, re-read chats after
dropping their queues, warn on every failed request, give each request
a 30-second limit, and treat only a stopped build as stopped: a failed
start gets a stop, and a failed stop warns. Bound both cleanup steps,
and say in the step summary when a review is stopped at the wait.

Cover each warning branch, failed builds, and paused chats in the test,
and keep transitions in the fake API for two reads, so the cases show
that the script waits for them.

Refs DOCS-1478
@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of 5a270bc. The author replied to every round 1 finding, and the commit addresses most of them. One finding is deferred with no ticket, so this round's review is blocked until it is resolved:

  • No bound on concurrent reviews across PRs (.github/workflows/doc-check.yaml:496). The reply says the cap is planned as a follow-up, once the post-PR concurrency is measured, but no issue is linked; DOCS-1478 is this PR's own ticket. This needs a human decision: link a ticket for the cap, or state that shipping without one is accepted, given that a burst of PRs past the model's slow-down point gives zero finished reviews.

The panel reviews the new commit, including the two contested findings and the acknowledged one, after this is resolved.

🤖 This review was automatically generated with Coder Agents.

@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of 5a270bc, which responds to round 1. Most round 1 findings are fixed, and reviewers confirmed the fixes in the code and tests. The carriage-return strip, the shared workspace lookup, the workspace restart by name, and leaving paused out of any_active are accepted as defended, and the concurrency cap is tracked in DOCS-1480. The end-to-end testing finding is reopened, because both dogfood runs ran 38970da, not this head. New findings: 3 P3, 9 P4, 3 Nit.

Out of scope (needs a ticket or explicit acceptance by a human):

  • .github/workflows/doc-check.yaml:304: the exact-name doc-check-pr-<N> lookup exists in Prepare the review workspace, delete-workspace, and doc-check-stop-review.sh, so a rename that misses one copy makes that lookup find no workspace.
  • .github/workflows/doc-check.yaml:220: the dispatch URL check accepts any owner/repo, but the prompt hard-codes coder/coder and the workspace name uses only the PR number, so a dispatch for another repository reviews the wrong PR.
  • .github/workflows/doc-check.yaml:50: a doc-check label added to a merged PR passes the job if and runs a review; if delete-workspace already deleted doc-check-pr-<N>, the review creates a new one that no later event deletes.

🤖 This review was automatically generated with Coder Agents.

Comment thread .github/workflows/doc-check.yaml Resolved
Comment thread .github/workflows/doc-check.yaml Resolved
Comment thread .github/workflows/doc-check.yaml Resolved
Comment thread .github/workflows/doc-check.yaml Resolved
Comment thread .github/workflows/doc-check.yaml Resolved
The stop script now ends within one time budget (STOP_BUDGET_SECONDS,
270 seconds by default) instead of per-loop sleep counts, and caps each
request at the time left. It also stops a workspace whose earlier stop
failed, re-reads a chat's queue after each delete, and looks up the
doc-check-pr-<number> workspace only for this repository's pull requests.

The workflow comments and summaries now match: any run in the group
cancels the one in progress, the label reviews every later push, the
timeout summary no longer repeats the wait or claims the stop worked, and
both cleanup steps say what their 6-minute limit covers.

Refs DOCS-1478
@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of bf1aa8b. The commit responds to every round 3 code finding except one. This round's review is blocked until that one gets a response:

  • The end-to-end runs in Testing ran 38970da, not the code under review (.github/workflows/doc-check.yaml:56). Its thread was resolved with no reply, and Testing still cites only runs 41729412601 and 41731313388 without naming the tested commit. Since then, 5a270bc and bf1aa8b rewrote stop_chat and stop_workspace, added the time budget, and moved the concurrency key to the PR URL; none of that has run on GitHub or the deployed Coder, and a manual run overlapping a pull request run has not run at any commit. Response needed: repeat the cancel and timeout runs at the current head (plus one manual run overlapping a pull request run) and name the SHA in Testing, or state in Testing that the runs tested 38970da and list what they do not cover.

The panel reviews bf1aa8b after this is resolved.

🤖 This review was automatically generated with Coder Agents.

@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of bf1aa8b, which responds to round 3. Reviewers checked the round 3 fixes in the code and tests and found them in place, and the end-to-end runs now ran this head (both report head_sha bf1aa8b). New findings: 2 P3, 11 P4, 3 Nit. The two P3s are about time: slow Coder requests during the chat phase can use the whole budget so the workspace is never stopped, and the 25-minute wait stops reviews that the decision log's own estimate says take longer even without load.

🤖 This review was automatically generated with Coder Agents.

Comment thread .github/workflows/doc-check.yaml Resolved
The chats now get a deadline at a third of the budget, enforced on every
request, queue read, and poll, so slow chat requests no longer use the
time the workspace stop needs. Every workspace gets at least one read,
and sleeps are cut to the time left.

The script also reads deleted workspaces, drops a full queue of 20
without a false warning, says when it could not read a chat or ran out
of time instead of claiming the chat keeps running, and words the
workspace warnings for the states that reach them. New cases cover two
matching reviews, a full queue, a deleted workspace, a mixed-case
repository URL, and the request time limit and its floor.

Refs DOCS-1478
@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of ccecdfa, which responds to round 5. Reviewers checked the round 5 fixes in the code and tests and found them in place: per-phase deadlines, the first workspace read after the deadline, include_deleted, the full-queue read, and the new cases. The 25-minute wait tradeoff is accepted now that the decision log states the count. New findings: 1 P3, 10 P4, 3 Nit. Most are about the new deadline code: a regression where a second review chat is never interrupted, branches with no test, and comments and warnings that describe the deadline incorrectly.

🤖 This review was automatically generated with Coder Agents.

Comment thread .github/workflows/doc-check.yaml Resolved
Every review chat now gets its queue dropped and a first interrupt before
the script waits on any of them, so a first chat that uses up the chat
phase no longer leaves a second one running. The chat phase's deadline
limits only the waiting.

A poll now starts only when a full poll interval and the read after it
fit in the phase, so sleeps are never cut short and each poll reads after
it sleeps. The phase boundaries sit together in main under a named
constant, the warnings say what a failed queue drop leads to, and the
start-after-stop warning names the likely cause.

Refs DOCS-1478
@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of af47edc, which responds to round 6. Reviewers checked the round 6 fixes and found them in place: every review chat gets its queue dropped and a first interrupt before any wait, sleep_if_time gates the polls in both loops, the round 6 header, comment, and warning fixes are in, and the two end-to-end runs in Testing ran this head. New findings: 1 P2, 1 P3, 12 P4, 2 Nit. The P2 is the rewritten chat-budget case, which fails on unchanged code under load, so the Testing sentence about 8 copies does not hold.

Three of the new findings are branches that a one-line mutation removes with the suite still passing; rounds 1, 5, and 6 had findings of the same kind. Deleting or inverting each new condition and running the suite before pushing would find these.

🤖 This review was automatically generated with Coder Agents.

Each phase, chats then workspaces, now runs one sweep function on every
item before it waits on any, then sweeps the items that have not settled
until the phase ends. Workspaces no longer wait on each other, every
sweep drops a busy chat's queue before it interrupts, and a chat or
workspace whose read fails is read again instead of dropped.

The tests now run on a virtual clock that the fake sleep, date, and slow
requests share, so the timing cases are deterministic and take no real
time. The fake can fail a path on chosen calls and queue a message when
a chat is interrupted, and new cases cover a read that fails and then
recovers, a read lost after a success, two workspaces, and the poll rule.

Refs DOCS-1478
@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of 10eb134, which responds to round 7 by replacing the two stop loops with run_phase and per-item sweeps and giving the test fake a virtual clock. Reviewers confirmed the round 7 fixes: chat-budget no longer depends on machine speed (8 copies of the suite passed at load average 134), every chat's queue is dropped before its first interrupt, workspaces no longer wait on each other, the poll counts are gone, the new read-recovery cases fail under the breaks they target, and the two end-to-end runs in Testing ran this head. New findings: 1 P2, 2 P3, 10 P4, 4 Nit.

This is the third round in a row where a Testing claim did not match what reviewers measured (the 8-copy claim in round 7, the mutation claim now). Listing only the checks that were run, with their results, would let a reader rely on the section.

Notes:

  • .github/workflows/doc-check.yaml:506: until the reviews started before this PR end, the PR's own timeout run made 6 model steps in 25 minutes against a median of 29, so most runs right after merge may post nothing. Running the stop script once for each open PR with a doc-check chat at merge time would remove that load; a human can decide whether that is worth doing.

🤖 This review was automatically generated with Coder Agents.

A failed workspace stop request is now retried after a new read instead
of settling the workspace, so a 409 from a build that started after the
read, or a transient error, no longer leaves the workspace running. The
warning moves to the end of the phase, and a build that starts after a
failed request is reported as a build that did not finish.

A failed interrupt no longer warns, because a chat that ends its turn
after the read answers 409 and the next sweep reads it again. A queue
delete that fails is retried after a new read, because chatd answers 404
for a message it promoted after the read. The busy statuses are written
once, and the after-stop checks are one block.

The fake now answers a stop with 409 while a build is in progress, and
can drop a queued message after a read. New cases cover a failed queue
read, a queue past 20 deletes, a promoted message, a failed interrupt, a
failed or retried stop, each build status, and a workspace read lost
after a success. The suite stops a case that polls past 3000 virtual
seconds, so its real-time limit is only a last resort.

Refs DOCS-1478
@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review

@coder-agents-review

Copy link
Copy Markdown
Contributor

@nickvigilante ⛔ This review has reached its per-chat spend limit ($106.48 / $100.00). Further review rounds are paused.

To raise the limit and continue, comment:

/coder-agents-review set-spend-limit:150

This is a per-chat budget, separate from any account-level usage limit.

🤖 Managed by Coder Agents.

@nickvigilante

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review set-spend-limit:250

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of f038096, which responds to round 8. Reviewers confirmed the round 8 fixes with targeted cases and mutations: failed stops are retried, the fake answers a stop during a build with an error and each in-progress status has a case, the busy status set is written once, clear_queue moves past a promoted message, the hang guard runs on the virtual clock (two-workspaces took 10 seconds on one core), two reviewers reran 20 and 6 of the 88 breaks in the Testing table and each failed its case, and the two end-to-end runs ran this head. New findings: 1 P3, 5 P4, 4 Nit. Most are about what the step log shows when a request fails repeatedly or a stop's result is unknown.

Notes:

  • .github/scripts/doc-check-stop-review.sh:109: swapping report_workspace's first two branches, or moving the qid == failed check below the last-pass break, passes the suite; each changes only which warning prints, and Testing says the 88 breaks are not every edit.
  • .github/workflows/doc-check.yaml:68: the review job and delete-workspace share a concurrency group, so with cancel-in-progress: true a review job that starts after a merge (from the doc-check label or a dispatch) cancels a running delete. This needs the merged-PR review that the out-of-scope label item already allows, and fixing that item also fixes this.

🤖 This review was automatically generated with Coder Agents.

Addresses the round 9 coder-agents-review findings on #31720.

- Warn once per chat about its queue. GitHub keeps only the first 10
  warning annotations of a step, and a busy chat's queue is read again on
  every sweep, so repeats hid the warnings that a review may still run.
- Warn "Could not interrupt chat <id>" for a chat that still reads as
  running after its last interrupt failed, instead of reporting it as a
  chat that ignored interrupts.
- Count a stop build that follows a failed stop request as this run's
  stop, because a request that timed out may have been accepted. Print
  the workspace's status when a failed request is followed by a stopped
  workspace, so the log no longer ends at "Stopping workspace".
- Drop the queue of a chat in error. chatd keeps it, and the next message
  sent to the chat runs the oldest queued one first.
- Make the fake API fail a workspace ID it has no entry for, as coderd
  does, and fix the test comments the review flagged.

DOCS-1478
@nickvigilante
nickvigilante temporarily deployed to main November 9, 2026 19:11 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 9, 2026 19:13 — with GitHub Actions Inactive
@nickvigilante
nickvigilante marked this pull request as ready for review November 9, 2026 19:14
@nickvigilante
nickvigilante requested a review from bpmct November 9, 2026 19:14
@nickvigilante
nickvigilante temporarily deployed to main November 9, 2026 19:14 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 9, 2026 19:41 — with GitHub Actions Inactive
nickvigilante added a commit that referenced this pull request Nov 12, 2026
…ment (#31994)

The doc-check agent posted the raw JSON of its review request as the
summary comment on #31385
([comment](#31385 (comment))),
so GitHub showed `{"commit_id": ..., "event": "COMMENT", "body": "##
Documentation Check\n..."}` with escaped newlines. The inline review on
that run posted correctly. Of the latest summary comments on 42 PRs
reviewed since 2026-10-25, this is the only malformed one.

This changes `.claude/skills/doc-check/output.md` so the agent:

- writes the review request to `review.json` and leaves the review
`body` empty, which matches what every other run does;
- writes the summary to its own `summary.md`, never reuses `review.json`
for it, and posts or edits it with exact `gh api ... -F
body=@summary.md` commands;
- checks, when it reads the comment back, that the body starts with `##
Documentation Check` and ends with the marker, and fixes it with `PATCH`
if not.

### Follow-up: workflow guard (not in this PR yet)

A workflow job that fails the run when the sticky summary comment does
not start with `## Documentation Check` is ready, but the bot that
opened this PR can't push changes under `.github/workflows/`. The patch
adds a `check-summary-comment` job (`needs: doc-check`, `pull-requests:
read`) at the end of `doc-check.yaml`, after `delete-workspace`, so it
stays clear of the hunks #31720 changes. It passed `actionlint` (with
shellcheck) and `zizmor --offline` with no findings, and a dry run
against live comments failed on #31385 and passed on #31896 and #31934.

Related open PRs: #31649 (draft) also edits `output.md`, and #31720
edits `doc-check.yaml`. Whichever lands second may need a small rebase.


Supersedes #31966, reopened under a human author for attribution.

Co-authored-by: coder[bot] <137810501+coder[bot]@users.noreply.github.com>
@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 18:31 — with GitHub Actions Inactive
bpmct added a commit that referenced this pull request Nov 13, 2026
…run ends (#32185)

## Summary

doc-check reused one chat for every push to a PR, and a review kept
running in Coder after its job stopped waiting for it.
This PR starts a new chat for each run, waits up to 5 hours for it, and
stops the review and its workspace when the run ends for any reason.

On dogfood at 2026-11-13 17:28 UTC, 58 doc-check review chats and 45 of
their sub-agents had status `running`, while 5 workflow runs were in
progress and 1 was queued.
34 of the 58 chats were created before that day, and the oldest on
2026-09-29.
The count covers only chats shared with me, so it is a lower bound.

This combines the workflow half of #32061 with the stop script from
#31720, and changes no skill file.

## Changes

- **New chat per run.**
`force-new-chat` replaces the one chat per PR, so a re-run no longer
carries every earlier turn on the PR.
The prompt gets a `## Previous run` section with the `doc-check-trace`
of the latest finished review.
- **One workspace per PR.**
The job finds `doc-check-pr-<N>`, wakes and starts it, and passes its ID
to the chat.
  On the first run the chat creates the workspace under that name.
- **A started review finishes; pushes are debounced.**
The review job's concurrency group is keyed on the PR URL with
`cancel-in-progress: false`. GitHub keeps one pending run per group and
replaces it on each push, so the next review covers the latest head.
A `debounce` job sleeps 5 minutes on `synchronize`, in its own group
with `cancel-in-progress: true`, so a burst of pushes starts one review.
Other triggers do not wait.
Merging the PR still cancels a running review (the `delete-workspace`
job keeps `cancel-in-progress: true`), then deletes the workspace.
- **Every run stops its review.**
`.github/scripts/doc-check-stop-review.sh` runs before the review and in
a final `always()` step.
It drops queued messages, interrupts the chat and its sub-agents, and
stops the workspace.
- **5-hour wait.**
`wait-timeout-seconds` goes from 2400 to 18000, and the job timeout from
50 to 330 minutes.
The job now stays in its concurrency group for the whole review, so the
next run for the PR waits for it.
- **Previous trace lookup.**
`.github/scripts/doc-check-previous-trace.sh` reads the 5 newest review
chats for the PR and prints the first trace with an `outcome:` line.
It reads more than one chat because a cancelled run's chat has no trace.
  A failed lookup costs only that context and never fails the job.
- The `GITHUB_OUTPUT` delimiter for the prompt is random, because the
trace is model output.

### Behavior changes to check

| Before | After | Why |
|--------|-------|-----|
| A review past the wait ended the job with a green notice and kept
running (#32097) | The last step stops the review, and the job fails |
Nothing watches a review after the job ends |
| A review kept running after its PR merged | The merge cancels the run
| The result has no reader |
| The `doc-check` label forced one review, on the `labeled` event | The
label forces a review on every push until it is removed | The push run
would otherwise skip on paths with no docs surface |
| Runs for one PR queued | A started review finishes; pushes during it
collapse into one pending run for the latest head, after a 5-minute
debounce | Cancelling threw away prefill and starved busy PRs of a
verdict |

## Testing

Static checks, all passing: `actionlint`, `zizmor`, `shellcheck
--external-sources`, and `shfmt -d`.

Script tests, run locally:

- `doc-check-stop-review.test.sh`: 48 cases against a fake API.
- `doc-check-previous-trace.test.sh`: 22 cases, including CRLF in the
trace text and in the PR URL.
- Mutation check on the trace script: 25 one-line breaks, and the suite
failed on 24.
  The survivor was a redundant trim, which I removed.
- The trace extraction ran against 12 real finished doc-check chats on
dogfood and returned a trace for all 12 (750 to 3,500 characters).

End to end on dogfood, on this PR, 2026-11-13 UTC:

| Run | Trigger | Result |
|-----|---------|--------|
| [1](https://github.com/coder/coder/actions/runs/43474029515) |
dispatch, 17:33 | No workspace and no earlier chat. New chat created the
workspace `doc-check-pr-32185` and had 2 running sub-agents at 17:48:23.
|
| [2](https://github.com/coder/coder/actions/runs/43476429715) |
dispatch, 17:48:25 | Cancelled run 1 at 17:48:27. Run 1's last step
interrupted the chat and both sub-agents and stopped the workspace by
17:48:51. Run 2 then started the same workspace, opened a second chat in
it, finished clean at 17:52:54, and stopped the workspace. |
| [3](https://github.com/coder/coder/actions/runs/43477511866) |
dispatch, 17:55:08 | Cancelled by run 4 while the workspace was
starting. Its last step stopped the workspace mid-start. No chat was
created. |
| [4](https://github.com/coder/coder/actions/runs/43477572348) |
`ready_for_review`, 17:55:31 | Waited for run 3 to end, then opened a
third chat in the same workspace. That chat's first message carries run
2's trace under `## Previous run`. |

Not covered:

- The 5-hour timeout path has not run live.
The fake-API tests cover the script, and the summary text is a few lines
of shell.
- The merge path (`delete-workspace` cancelling a review) has not run
live.
- No CI job runs the two `.test.sh` suites.
The workflow runs on `ubuntu-latest` only, and the scripts were not run
on macOS or Windows.

## Follow-ups, not in this PR

- Skip a run when the head SHA or the PR diff matches the last reviewed
one.
- Set a status on the head SHA from the trace.
- Cancel a review when its PR closes without merging.

Supersedes #31720 and the workflow half of #32061.

Linear: DOCS-1745

<details>
<summary>Plan and decision log</summary>

This is PR 1 of a 5-part stack from the design doc "Design: reliable,
low-cost doc-check on Solstice".

1. Workflow base: new chat per run, previous trace, workspace ID, 5-hour
wait, cancel-in-progress, and the stop script. **This PR.**
2. Skip in Actions: same head SHA, same PR diff, and a delta since the
last reviewed SHA with no docs surface.
3. Visible outcome: a status on the head SHA from the trace, for clean,
findings, error, and skipped.
4. Packet and lint in Actions: PR data, diffs, open threads, and
deterministic check results, written once.
5. Skill: scope to wrong, missing, and broken, tiers, a sub-agent cap,
and one long wait. Gated by an eval.

Decisions in this PR:

- **Why this first.**
A re-run's first prompt was 1.01 times the previous run's last prompt,
so each push re-sent the whole history of the PR.
A new chat per run removes that, and stopping stale reviews removes load
that no job is watching.
- **5 hours, not 25 or 40 minutes.**
Reviews on the local model took a median of 61 minutes when clean and
247 minutes with findings (2026-11-04 to 11-11).
  A shorter wait would stop most reviews with findings before they post.
With `cancel-in-progress`, a new push still ends the review of the old
head at once.
- **A timeout fails the job.**
  This reverses #32097, where a timeout was a notice.
  A green job with no review reads as a clean result.
- **The stop runs twice.**
The first step catches a review left by a run whose last step did not
finish, for example a runner that died.
- **The trace lookup is a script with tests, not inline YAML.**
It parses model output, and it has to handle chats with no trace, a
trace with no outcome, and CRLF.
- **The trace is passed as a record, not as instructions.**
  The prompt says so, and asks for a review of the whole PR.
A delta review needs the skip logic and the ledger from later PRs in the
stack.
- **No skill change.**
  The skill changes are gated by an eval that does not exist yet.

</details>

Follow-up commit `64a09655a1` (finish-then-rerun and debounce) generated
by Coder Agents on behalf of @bpmct.

_Generated by Coder Agents on behalf of @nickvigilante._

---------

Co-authored-by: Ben Potter <me@bpmct.net>

This branch was previously deployed

1 inactive deployment
main — bf2c4ef6 Deployed Nov 13, 2026 by nickvigilante via cla #256900
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant