Skip to content

ci(.github): start a new doc-check chat per run and stop it when the run ends - #32185

Merged
bpmct merged 3 commits into
mainfrom
vigilante/docs-1745-doc-check-start-a-new-chat-per-run-wait
Nov 13, 2026
Merged

bpmct merged 3 commits into
mainfrom
vigilante/docs-1745-doc-check-start-a-new-chat-per-run-wait

Conversation

@nickvigilante

@nickvigilante nickvigilante commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

doc-check reused one chat for every push to a PR, and a review kept running in Coder after its job stopped waiting for it.
This PR starts a new chat for each run, waits up to 5 hours for it, and stops the review and its workspace when the run ends for any reason.

On dogfood at 2026-11-13 17:28 UTC, 58 doc-check review chats and 45 of their sub-agents had status running, while 5 workflow runs were in progress and 1 was queued.
34 of the 58 chats were created before that day, and the oldest on 2026-09-29.
The count covers only chats shared with me, so it is a lower bound.

This combines the workflow half of #32061 with the stop script from #31720, and changes no skill file.

Changes

  • New chat per run.
    force-new-chat replaces the one chat per PR, so a re-run no longer carries every earlier turn on the PR.
    The prompt gets a ## Previous run section with the doc-check-trace of the latest finished review.
  • One workspace per PR.
    The job finds doc-check-pr-<N>, wakes and starts it, and passes its ID to the chat.
    On the first run the chat creates the workspace under that name.
  • A started review finishes; pushes are debounced.
    The review job's concurrency group is keyed on the PR URL with cancel-in-progress: false. GitHub keeps one pending run per group and replaces it on each push, so the next review covers the latest head.
    A debounce job sleeps 5 minutes on synchronize, in its own group with cancel-in-progress: true, so a burst of pushes starts one review. Other triggers do not wait.
    Merging the PR still cancels a running review (the delete-workspace job keeps cancel-in-progress: true), then deletes the workspace.
  • Every run stops its review.
    .github/scripts/doc-check-stop-review.sh runs before the review and in a final always() step.
    It drops queued messages, interrupts the chat and its sub-agents, and stops the workspace.
  • 5-hour wait.
    wait-timeout-seconds goes from 2400 to 18000, and the job timeout from 50 to 330 minutes.
    The job now stays in its concurrency group for the whole review, so the next run for the PR waits for it.
  • Previous trace lookup.
    .github/scripts/doc-check-previous-trace.sh reads the 5 newest review chats for the PR and prints the first trace with an outcome: line.
    It reads more than one chat because a cancelled run's chat has no trace.
    A failed lookup costs only that context and never fails the job.
  • The GITHUB_OUTPUT delimiter for the prompt is random, because the trace is model output.

Behavior changes to check

Before After Why
A review past the wait ended the job with a green notice and kept running (#32097) The last step stops the review, and the job fails Nothing watches a review after the job ends
A review kept running after its PR merged The merge cancels the run The result has no reader
The doc-check label forced one review, on the labeled event The label forces a review on every push until it is removed The push run would otherwise skip on paths with no docs surface
Runs for one PR queued A started review finishes; pushes during it collapse into one pending run for the latest head, after a 5-minute debounce Cancelling threw away prefill and starved busy PRs of a verdict

Testing

Static checks, all passing: actionlint, zizmor, shellcheck --external-sources, and shfmt -d.

Script tests, run locally:

  • doc-check-stop-review.test.sh: 48 cases against a fake API.
  • doc-check-previous-trace.test.sh: 22 cases, including CRLF in the trace text and in the PR URL.
  • Mutation check on the trace script: 25 one-line breaks, and the suite failed on 24.
    The survivor was a redundant trim, which I removed.
  • The trace extraction ran against 12 real finished doc-check chats on dogfood and returned a trace for all 12 (750 to 3,500 characters).

End to end on dogfood, on this PR, 2026-11-13 UTC:

Run Trigger Result
1 dispatch, 17:33 No workspace and no earlier chat. New chat created the workspace doc-check-pr-32185 and had 2 running sub-agents at 17:48:23.
2 dispatch, 17:48:25 Cancelled run 1 at 17:48:27. Run 1's last step interrupted the chat and both sub-agents and stopped the workspace by 17:48:51. Run 2 then started the same workspace, opened a second chat in it, finished clean at 17:52:54, and stopped the workspace.
3 dispatch, 17:55:08 Cancelled by run 4 while the workspace was starting. Its last step stopped the workspace mid-start. No chat was created.
4 ready_for_review, 17:55:31 Waited for run 3 to end, then opened a third chat in the same workspace. That chat's first message carries run 2's trace under ## Previous run.

Not covered:

  • The 5-hour timeout path has not run live.
    The fake-API tests cover the script, and the summary text is a few lines of shell.
  • The merge path (delete-workspace cancelling a review) has not run live.
  • No CI job runs the two .test.sh suites.
    The workflow runs on ubuntu-latest only, and the scripts were not run on macOS or Windows.

Follow-ups, not in this PR

  • Skip a run when the head SHA or the PR diff matches the last reviewed one.
  • Set a status on the head SHA from the trace.
  • Cancel a review when its PR closes without merging.

Supersedes #31720 and the workflow half of #32061.

Linear: DOCS-1745

Plan and decision log

This is PR 1 of a 5-part stack from the design doc "Design: reliable, low-cost doc-check on Solstice".

  1. Workflow base: new chat per run, previous trace, workspace ID, 5-hour wait, cancel-in-progress, and the stop script. This PR.
  2. Skip in Actions: same head SHA, same PR diff, and a delta since the last reviewed SHA with no docs surface.
  3. Visible outcome: a status on the head SHA from the trace, for clean, findings, error, and skipped.
  4. Packet and lint in Actions: PR data, diffs, open threads, and deterministic check results, written once.
  5. Skill: scope to wrong, missing, and broken, tiers, a sub-agent cap, and one long wait. Gated by an eval.

Decisions in this PR:

  • Why this first.
    A re-run's first prompt was 1.01 times the previous run's last prompt, so each push re-sent the whole history of the PR.
    A new chat per run removes that, and stopping stale reviews removes load that no job is watching.
  • 5 hours, not 25 or 40 minutes.
    Reviews on the local model took a median of 61 minutes when clean and 247 minutes with findings (2026-11-04 to 11-11).
    A shorter wait would stop most reviews with findings before they post.
    With cancel-in-progress, a new push still ends the review of the old head at once.
  • A timeout fails the job.
    This reverses ci(.github/workflows): don't fail doc-check when the review outlives the wait #32097, where a timeout was a notice.
    A green job with no review reads as a clean result.
  • The stop runs twice.
    The first step catches a review left by a run whose last step did not finish, for example a runner that died.
  • The trace lookup is a script with tests, not inline YAML.
    It parses model output, and it has to handle chats with no trace, a trace with no outcome, and CRLF.
  • The trace is passed as a record, not as instructions.
    The prompt says so, and asks for a review of the whole PR.
    A delta review needs the skip logic and the ledger from later PRs in the stack.
  • No skill change.
    The skill changes are gated by an eval that does not exist yet.

Follow-up commit 64a09655a1 (finish-then-rerun and debounce) generated by Coder Agents on behalf of @bpmct.

Generated by Coder Agents on behalf of @nickvigilante.

…run ends

doc-check reused one chat for every push to a PR, so a re-run sent the
whole earlier review again, and a review kept running after its job
stopped waiting. On 2026-11-13, 58 review chats and 45 of their
sub-agents were running on the shared model while 5 workflow runs were
in progress.

Each run now starts a new chat in the PR's review workspace and gets the
trace of the latest finished review in its prompt. A new run cancels the
one in progress, and every run stops its review and workspace when it
ends. The wait is 5 hours, and a review that runs past it is stopped
and fails the job.

This combines the workflow half of #32061 with #31720.

Linear: DOCS-1745

Co-authored-by: Ben Potter <me@bpmct.net>
@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 17:33 — with GitHub Actions Inactive
@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 17:33 — with GitHub Actions Inactive
@linear-code

linear-code Bot commented Nov 13, 2026

Copy link
Copy Markdown

DOCS-1745

@nickvigilante
nickvigilante marked this pull request as ready for review November 13, 2026 17:55
@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 17:55 — with GitHub Actions Inactive
@nickvigilante
nickvigilante requested a review from bpmct November 13, 2026 17:55
@nickvigilante
nickvigilante temporarily deployed to main November 13, 2026 18:00 — with GitHub Actions Inactive
A push no longer cancels a review in progress. GitHub keeps one pending
run per PR and replaces it on each push, so the next review covers the
latest head. A 5-minute debounce job, which each push cancels, collapses
a burst of pushes into one review. A merge still cancels the running
review through the delete-workspace job.

Generated by Coder Agents on behalf of @bpmct.
@bpmct
bpmct temporarily deployed to main November 13, 2026 20:55 — with GitHub Actions Inactive
@bpmct
bpmct temporarily deployed to main November 13, 2026 20:55 — with GitHub Actions Inactive
@bpmct
bpmct enabled auto-merge (squash) November 13, 2026 20:58
@bpmct
bpmct temporarily deployed to main November 13, 2026 20:58 — with GitHub Actions Inactive
@bpmct
bpmct merged commit dc55633 into main Nov 13, 2026
30 checks passed
@bpmct
bpmct deleted the vigilante/docs-1745-doc-check-start-a-new-chat-per-run-wait branch November 13, 2026 21:14
@bpmct
bpmct temporarily deployed to main November 13, 2026 21:14 — with GitHub Actions Inactive
@bpmct
bpmct temporarily deployed to docs-preview November 13, 2026 21:14 — with GitHub Actions Inactive
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 13, 2026

This branch was previously deployed

2 inactive deployments
docs-preview — 2e021b45 Deployed Nov 13, 2026 by bpmct via cleanup #60
main — 2e021b45 Deployed Nov 13, 2026 by bpmct via cla #257022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants