Skip to content

feat(coderd): protect pending jobs only while the provisioner queue is moving - #32099

Merged
dylanhuff-at-coder merged 9 commits into
mainfrom
dylan/reaper-queue-progress-check
Nov 16, 2026
Merged

dylanhuff-at-coder merged 9 commits into
mainfrom
dylan/reaper-queue-progress-check

Conversation

@dylanhuff-at-coder

@dylanhuff-at-coder dylanhuff-at-coder commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Stops the pending job reaper from failing jobs in a queue that's still moving. This is an alternative to #31469 that came out of Steven's review asking whether 2h is enough, and it builds on George's work there.

A pending job past 30m is now only protected while a matching online provisioner is making progress, meaning it finished a job in the last 30m or is running one that's still being updated. A moving queue never gets reaped and a frozen one still does at 30m like today. There's no absolute cap.

No migration since it uses the existing provisioner_jobs_worker_id_organization_id_completed_at_idx. One tradeoff is that a stuck build whose provisioner keeps heartbeating counts as progress, so if every matching provisioner is stuck like that, the queue waits until that build is canceled.

Still a draft so we can talk it through with Spike and Steven.

Part of CORE-586

geokat and others added 5 commits October 28, 2026 16:10
Preserve pending jobs past the 30-minute timeout while an eligible
provisioner is online, including busy provisioners. Add a configurable
maximum pending age that defaults to two hours since creation.

Filter protected jobs before the candidate limit, recheck eligibility
under the job row lock, and reuse availability checks for jobs with
matching requirements.

Refs:
https://linear.app/codercom/issue/CORE-586
…s moving

A pending job past PendingJobDuration is now protected only while an
eligible online provisioner is making progress: it completed a job within
ProvisionerProgressWindow (30m) or is running a job that is not hung. A
slow but moving queue is never reaped, and a frozen one is.

The progress check is two EXISTS legs so each is a range scan on
provisioner_jobs_worker_id_organization_id_completed_at_idx. The locked
recheck in reapJob applies the same rule through the new
HasProvisionerDaemonMakingProgress query.

MaxPendingJobDuration moves from 2h to 24h and is now only a backstop for
jobs a moving pool never acquires.
Run the locked recheck against the real database for protected jobs, make
the long-build fixture start before the progress window, and cover both
progress legs in the candidate exclusion test. Also require provisioner
daemon read for HasProvisionerDaemonMakingProgress and reword the pending
reap message so it does not depend on the window length.
@linear-code

linear-code Bot commented Nov 12, 2026

Copy link
Copy Markdown

CORE-586

@dylanhuff-at-coder

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review Please limit the review to P0, P1, and P2 issues only.

@coder-agents-review

coder-agents-review Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-11-12 18:28 UTC by @dylanhuff-at-coder
Spend: $24.34 / $100.00

Review history
  • R1 (2026-11-12): 13 reviewers, 6 Nit, 6 P3, 5 P4, COMMENT. Review
  • R2 (2026-11-12): 16 reviewers, 6 Nit, 6 P3, 5 P4, APPROVE. Review

deep-review v0.13.0 | Round 2 | 6f22a88..205c8a4

Last posted: Round 2, 17 findings (6 P3, 5 P4, 6 Nit), APPROVE. Review

Finding inventory

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P3 Author accepted R2 (rare state, lasts one window; filtering needs a new error code) provisionerjobs.sql:366 Reaper-terminated hung job counts as recent completion, so a wedged daemon looks like progress for 30m R1 Netero Yes
CRF-2 P3 Author fixed (205c8a4) detector.go:45 ProvisionerStaleInterval doc comment still says online alone protects pending jobs R1 Netero Yes
CRF-3 P4 Author fixed (aa3d77c, function removed) detector.go:88 formatDuration minutes branch unreachable; prints "1 hours" R1 Netero Yes
CRF-4 P4 Author accepted R2 (one-line wrap; sibling error covered) detector.go:346 No test for HasProvisionerDaemonMakingProgress error in locked recheck R1 Netero Yes
CRF-5 Nit Author contested; panel closed R2 (1/1 accept) provisionerjobs.sql:414 HasProvisionerDaemonMakingProgress lacks By... suffix R1 Netero Yes
CRF-6 Note Dropped by orchestrator (merged into CRF-8, whose fix removes the duplication) provisionerjobs.sql:355 Progress rule duplicated in three places, no agreement test R1 Netero No
CRF-7 OOS Out of scope (body) 000364 migration:3 Index targets comment does not list reaper queries R1 Netero No
CRF-8 P3 Author contested; panel closed R2 (2/2 accept) detector.go:341 Locked recheck re-reads unlocked daemon state; second copy of progress rule adds order dependence, not safety R1 Ryosuke, Netero Yes
CRF-9 P3 Author contested; panel closed R2 (3/3 accept) provisionerjobs.sql:374 No test covers a hung running job; updated_at >= hung_since guard deletable in both queries R1 Bisky P3, Melody P4, Knov P4 Yes
CRF-10 P3 Author fixed (aa3d77c, reverted to main message) detector.go:79 Pending reap message covers no-online and online-not-progressing with one sentence R1 Mafuuu P3, Luffy Note Yes
CRF-11 Nit Author fixed (205c8a4) detector.go:33 "moving" and "making progress" used for the same condition R1 Gon Yes
CRF-12 P3 Author fixed (205c8a4) provisionerjobs.sql:415 HasProvisionerDaemonMakingProgress doc omits that it skips heartbeat/eligibility and is any-of R1 Gon Yes
CRF-13 Nit Author fixed (205c8a4) provisionerjobs.sql:355 Several new comments restate the code (sql:344, sql:355, detector.go:87, pending_test.go:64, detector_internal_test.go:81) R1 Gon Yes
CRF-14 Nit Author fixed (205c8a4) detector.go:323 Sync comment names the called query as a sync target R1 Gon Yes
CRF-15 Nit Dropped by orchestrator (matches sibling GetProvisionerJobsToBeReaped authorizer comment) dbauthz.go:6359 dbauthz comment restates the authorize call R1 Gon No
CRF-16 P4 Author fixed (205c8a4) pending_test.go:50 daemonProgress zero value gives cases that omit progress a progressing daemon, undocumented R1 Gon Yes
CRF-17 Nit Author fixed (205c8a4) pending_test.go:56 runningJob constant vs RunningLongBuild case names R1 Gon Yes
CRF-18 P4 Author fixed (205c8a4) pending_test.go:137 -35m literal instead of PendingJobDuration-derived age R1 Gon Yes
CRF-19 Nit Author fixed (205c8a4) provisionerjobs.sql:427 Same cutoff named provisioner_progress_since and progress_since R1 Gon Yes
CRF-20 P4 Author fixed (205c8a4) pending_test.go:185 Comment claims the candidate query excluded the job; assertion does not show that R1 Bisky Yes
CRF-21 Note Note (body) provisionerjobs.sql:355 Newly online daemon has no progress until first acquisition R1 Pariston No
CRF-22 Note Note (body) detector.go:37 24h cap value has no stated basis; wedged-heartbeating pool waits 24h instead of 2h R1 Luffy No
CRF-23 OOS Out of scope (body) dbauthz.go:5009 Sibling authorizer comment restates the authorize call R1 Gon No
CRF-24 OOS Out of scope (body) detector_internal_test.go:54 -35m literal on unchanged lines R1 Gon No
CRF-25 P3 Dropped by orchestrator (below requester's P0-P2 floor, R2) detector.go:33 No absolute cap: stuck-but-heartbeating pool or starved prebuilds keep jobs pending indefinitely R2 Knov P3, Luffy Note, Pariston Note No
CRF-26 P3 Dropped by orchestrator (below requester's P0-P2 floor, R2) provisionerjobs.sql:329 Large pending backlog pushes reaper query past JIT thresholds (1.4s JIT for 97ms query) R2 Killua No
CRF-27 P3 Dropped by orchestrator (below requester's P0-P2 floor, R2) detector.go:65 Pending reap message says "pending for 30 minutes" for jobs protected for hours R2 Leorio No
CRF-28 Nit Dropped by orchestrator (below requester's P0-P2 floor, R2) provisionerjobs.sql:330 pending_requirements comment first sentence restates CTE R2 Gon No
CRF-29 Nit Dropped by orchestrator (below requester's P0-P2 floor, R2) pending_test.go:120 OnlineLongQueue comment restates the row R2 Gon No
CRF-30 OOS Out of scope (body) provisionerd/runner/runner.go:472 No maximum build duration; heartbeating stuck build holds a daemon until canceled R2 Knov No

Contested and acknowledged

CRF-1 (P3, provisionerjobs.sql:366) - Reaper-terminated hung job counts as progress

  • Finding: The hung reap sets completed_at = now on a job whose worker_id is the wedged daemon, so the completed leg treats that daemon as making progress for 30 minutes; proposed excluding reaper-terminated jobs (distinct error_code).
  • Author defense (R2, acknowledged): Needs a daemon online while its job stops updating, lasts one window, and filtering needs a new error code or error-text matching, which is too much for this PR.

CRF-4 (P4, detector.go:346) - Untested progress-query error in recheck

  • Finding: No test covers the error return from HasProvisionerDaemonMakingProgress.
  • Author defense (R2, acknowledged): One-line error wrap; sibling lookup error already covered.

CRF-5 (Nit, provisionerjobs.sql:414) - Query name lacks By... suffix

  • Finding: Rename HasProvisionerDaemonMakingProgress to name its filter column.
  • Author defense (R2, contested): Has... reads fine and the parameter says it takes daemon IDs.

CRF-8 (P3, detector.go:341) - Locked recheck duplicates progress rule over unlocked state

  • Finding: The recheck under the job row lock re-reads unlocked daemon and job state; proposed deleting the daemon/progress recheck and HasProvisionerDaemonMakingProgress.
  • Author defense (R2, contested): Not airtight, but it narrows the window between picking and locking the job and catches a hung job reaped earlier in the same tick.

CRF-9 (P3, provisionerjobs.sql:374) - Hung running-job guard untested

  • Finding: No test covers a running job older than HungJobDuration; the updated_at >= @hung_since guard can be removed from both queries with tests passing.
  • Author defense (R2, contested): Without the guard a hung job counts only until the hung reaper completes it, so it changes the outcome for about one window and does not hold the queue indefinitely.

Round log

Round 1

Full panel (round 1). Netero first pass: 2 P3, 2 P4, 1 Nit, 2 Note, no P0, so panel ran. Law not run (effective additions 294). Panel: ging-go, knuckle, killua, ryosuke, kurapika, melody, luffy (specialists); pariston, mafuuu (core); bisky, gon, leorio (pool); knov (wildcard). Multi-domain target 9-12; 12 + wildcard, all specialists triggered by Go, SQL, dbauthz, and feature change. Gon's comment-restatement findings (rated P2) downgraded to Nit as project-rule violations with working code and consolidated into CRF-13. CRF-9 linked to CRF-1: the proposed tick-based hung-job test row is order dependent while reaper-terminated hung jobs count as progress (Netero reproduced 4/6 protected). Ging-go, knuckle, killua, kurapika, leorio: no findings; knuckle and killua EXPLAIN runs support the no-migration index claim. 6 P3, 5 P4, 6 Nit posted; 2 dropped; 2 Note (body); 3 OOS. Reviewed against 79d25dd..5512b0c.

CRF-1 and CRF-4 author accepted (R2)

  • Author accepted: CRF-1 accepted by the author as lasting one ProvisionerProgressWindow in a rare state; Ryosuke, Melody, Bisky, and Knov relied on that bound when closing CRF-8 and CRF-9. CRF-4 accepted as a one-line wrap; Netero (raiser) confirmed the branch is still uncovered (detector.go:311-313, count 0) and did not re-raise. No ticket for either; review body asks for human acceptance.

CRF-5, CRF-8, CRF-9 panel closure (R2)

  • CRF-5 (1/1): Netero, the original agent, accepted: no behavior change, and the other Has... query (HasTemplateVersionsUsingCachedModuleFileInOrg) also lacks a By... suffix.
  • CRF-8 (2/2): Ryosuke and Netero, both original agents, accepted: the recheck narrows the pick-to-lock window; its same-tick effect is the CRF-1 consequence the author accepted; remaining cost is P3 drift risk between two copies, covered by tests that run both paths.
  • CRF-9 (3/3): Bisky, Melody, Knov, all original agents, accepted: the running-leg guard and the hung reap leg are complements on @hung_since, so removing the guard changes the outcome by at most a tick before the hung job is reaped; it cannot hold the queue.

Round 2 update

Restructure round (PR rebased standalone against main, MaxPendingJobDuration removed, description states no absolute cap). Churn guard PROCEED: 12 addressed, 2 acknowledged, 3 contested. Netero (reused agent; first message produced no file, re-prompted) found no new findings. Full panel: ging-go, knuckle, killua, kurapika, luffy (specialists), pariston, mafuuu (core), gon, leorio (floor, restructure), hisoka, komugi (pool), razor (wildcard), plus contested raisers ryosuke, melody, bisky, knov via their R1 agents. Requester asked in PR comments for P0-P2 only; no reviewer raised P0-P2. Gon's P2 comment restatements downgraded to Nit (convention, code works). New: 3 P3, 2 Nit, all below floor and not posted; 1 OOS. Reviewed against 6f22a88..205c8a4aaf77.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR protects pending jobs past 30 minutes only while an eligible online provisioner has completed a job in the last 30 minutes or is running a job that is still updated, and raises the absolute cap from 2 hours to 24 hours. Findings: 6 P3, 5 P4, 6 Nit. EXPLAIN runs on seeded PostgreSQL 13 databases showed both progress legs use provisioner_jobs_worker_id_organization_id_completed_at_idx, which supports the no-migration claim.

Out of scope (needs a ticket or explicit acceptance by a human):

  • coderd/database/migrations/000364_optimize_getprovisionerdaemonswithstatusbyorganization_query.up.sql:3: the index targets: comment lists only GetProvisionerDaemonsWithStatusByOrganization, so someone pruning indexes by that comment would not see that the reaper queries now depend on it.
  • coderd/database/dbauthz/dbauthz.go:5009: the GetProvisionerJobsToBeReaped authorizer comment restates the authorize call below it.
  • coderd/jobreaper/detector_internal_test.go:54: the -35 * time.Minute literal for "past PendingJobDuration" is also on unchanged lines here.

Notes:

  • A daemon that comes online after the pool was down has no progress until its first acquisition, so a reaper tick in that window reaps matching jobs older than 30 minutes, where the heartbeat alone protected them before. The window is the time to first acquisition.
  • The 24 hour value has no stated basis beyond "backstop". When every eligible daemon runs a build that keeps updating but never finishes, the queue behind it now gets an error after 24 hours instead of 2. That number needs a human decision.

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/jobreaper/detector.go Resolved
Comment thread coderd/jobreaper/detector.go Resolved
Comment thread coderd/jobreaper/detector.go Resolved
Comment thread coderd/jobreaper/pending_test.go Resolved
Comment thread coderd/jobreaper/pending_test.go Resolved
Comment thread coderd/jobreaper/pending_test.go Resolved
Comment thread coderd/jobreaper/pending_test.go Resolved
Drop the absolute pending age cap, so a pending job stays protected for as
long as an eligible provisioner is online and making progress, and restore
main's reap messages and reapJob signature. The progress check, its locked
recheck, and the tests that pin them are unchanged.
Use "making progress" consistently, document the stuck-build tradeoff and
the caller contract of HasProvisionerDaemonMakingProgress, and rename its
progress parameter to match GetProvisionerJobsToBeReaped. Tests derive job
ages from PendingJobDuration, and the canceled-before-acquisition case now
pins the pending status condition of the protection clause.
@dylanhuff-at-coder
dylanhuff-at-coder changed the base branch from george/core-586/fix/protect-pending-provisioner-jobs to main November 12, 2026 18:26
@dylanhuff-at-coder
dylanhuff-at-coder temporarily deployed to main November 12, 2026 18:26 — with GitHub Actions Inactive
@dylanhuff-at-coder
dylanhuff-at-coder temporarily deployed to main November 12, 2026 18:26 — with GitHub Actions Inactive
@dylanhuff-at-coder

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review Please limit the review to P0, P1, and P2 issues only.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This round reviewed the standalone version that protects pending jobs only while a matching online provisioner is making progress and has no absolute pending cap. As requested, it covers P0, P1, and P2 only, and there are none; 3 P3 and 2 Nit findings were not posted.

The round 1 fixes are in place. The defenses on the query name, the locked recheck, and the hung running-job test were accepted. Two findings were acknowledged without a fix or ticket (a reaper-terminated hung job counting as progress, and the untested progress-query error branch); that acceptance needs to come from a human.

Out of scope (needs a ticket or explicit acceptance by a human):

  • provisionerd/runner/runner.go:472: builds have no maximum duration, so a build that keeps heartbeating holds its provisioner until someone cancels it; with no pending cap, the queue behind it waits for that cancel too.

🤖 This review was automatically generated with Coder Agents.

@dylanhuff-at-coder
dylanhuff-at-coder marked this pull request as ready for review November 12, 2026 19:06
@dylanhuff-at-coder
dylanhuff-at-coder temporarily deployed to main November 12, 2026 19:06 — with GitHub Actions Inactive
@coderagents

coderagents Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Documentation Check

Blocking

  • docs/install/operate/provisioners/manage-provisioner-jobs.md (open) - Line 8 says a pending job can "hang indefinitely" with no mention that Coder's reaper now auto-fails a pending job after 30 minutes unless a matching, online provisioner is making progress (completed a job in the last 30 minutes, or is actively updating a running job within the last 5 minutes). (Newcomer)
  • docs/install/operate/provisioners/manage-provisioner-jobs.md (open) - The "stuck in Pending for too long" bullet (line 62-63) lists only misconfigured tags or unavailable provisioners as causes, and presents cancellation as one option among several; it doesn't mention that a pending job behind an online, "making progress" provisioner pool now has no absolute age cap and that cancellation is the only way to clear it. (Cartographer)

These two findings are about docs/install/operate/provisioners/manage-provisioner-jobs.md, a page this PR does not touch, so they can't be posted as inline review comments; see the finding blocks below for evidence and suggested replacement text.

Related open PRs

  • docs: add provisioner metrics and a metrics signpost page #31977 ("docs: add provisioner metrics and a metrics signpost page") edits docs/install/operate/provisioners/manage-provisioner-jobs.md and docs/install/operate/provisioners/index.md, the two files with findings above. It appends after the existing content and doesn't touch the lines these findings target, but a reviewer picking up both PRs should apply this review's fixes after rebasing past whichever merges first.
  • fix: avoid premature reaping of pending provisioner jobs #31469 ("fix: avoid premature reaping of pending provisioner jobs") is the earlier, alternative approach to the same problem this PR solves instead; it touches no docs and doesn't conflict with any claim here.
Findings for jobreaper
- id: learner-1
  persona: learner
  class: missing
  blocking: true
  file: docs/install/operate/provisioners/manage-provisioner-jobs.md
  line: 8
  quote: "While most jobs complete smoothly, issues with templates, cloud resources, or misconfigured provisioners can cause jobs to fail or hang indefinitely (these are in a `Pending` state)."
  evidence:
    - coderd/jobreaper/detector.go:26-29
    - coderd/jobreaper/detector.go:284-319
  rule: null
  problem: The page teaches that pending jobs just hang, with no mention that Coder itself automatically fails a pending job after 30 minutes unless a matching, online, "making progress" provisioner exists; a reader troubleshooting a stuck queue would not know some of their pending jobs are being auto-reaped while others, behind a working but backlogged pool, are protected indefinitely and will never auto-clear no matter how old they get.
  fix: "Add a note near this paragraph (and in the 'When to cancel provisioner jobs' and 'Troubleshoot provisioner jobs' sections) explaining that Coder's job reaper automatically fails a pending job older than 30 minutes unless an eligible, recently-heartbeating provisioner for that job's organization/type/tags has completed a job in the last 30 minutes or is actively updating a running job within the last 5 minutes; otherwise the job can remain pending indefinitely as long as such a provisioner keeps making progress."
  verified: true

- id: coverage-1
  persona: coverage
  class: missing
  blocking: true
  file: docs/install/operate/provisioners/manage-provisioner-jobs.md
  line: 62
  quote: "- It has been stuck in **Pending** for too long.\n  This can be due to misconfigured tags or unavailable provisioners."
  evidence:
    - coderd/jobreaper/detector.go:284-319
    - coderd/database/queries/provisionerjobs.sql:328-407
    - "coderd/jobreaper/detector.go:388-394 (confirms this is a timing-behavior change only; the stored error text/ErrorCode on a reaped job is unchanged)"
  rule: null
  problem: Before this PR, every pending provisioner job older than 30 minutes was reaped unconditionally, so a stuck Pending job always auto-resolved within that window even without intervention. This PR makes that reap conditional on provisioner state: a pending job now stays Pending indefinitely, with no absolute age cap, whenever a matching, online provisioner keeps completing other jobs or keeps updating a running job. The bullet lists only "misconfigured tags or unavailable provisioners" and presents cancellation as one option among several remedies, but doesn't mention the new scenario (a correctly tagged, online, busy-but-progressing provisioner pool) or that canceling is now the only way to clear such a job, since the reaper no longer times it out automatically in that case.
  fix: "Expand the bullet to cover the new cause and state that cancellation is now the only remedy for it, e.g.: It has been stuck in Pending for too long. This can be due to misconfigured tags, unavailable provisioners, or a busy provisioner pool that keeps making progress on other jobs. Coder only auto-fails a pending job once no matching online provisioner has made progress for a while, so a job behind an otherwise-healthy pool can stay pending indefinitely until you cancel it."
  verified: true

- id: accuracy-1
  persona: accuracy
  class: suggestion
  blocking: false
  file: coderd/database/queries/provisionerjobs.sql
  line: 354
  quote: "-- scan on provisioner_jobs_worker_id_organization_id_completed_at_idx."
  evidence:
    - coderd/database/migrations/000364_optimize_getprovisionerdaemonswithstatusbyorganization_query.up.sql:1
    - coderd/database/queries/provisionerjobs.sql:328-407
  rule: null
  problem: The comment and the PR description both assert the two progress EXISTS legs stay index range scans on provisioner_jobs_worker_id_organization_id_completed_at_idx, but no test pins that plan; a future predicate or column-order change could silently fall back to a sequential scan without failing any test.
  fix: Add a planner-pinning regression test (e.g., in coderd/database/queries or a new query_test.go) that runs EXPLAIN on GetProvisionerJobsToBeReaped and HasProvisionerDaemonMakingProgress against a seeded table and asserts the plan uses provisioner_jobs_worker_id_organization_id_completed_at_idx, so a future edit that breaks the index match fails CI instead of only degrading production latency.
  verified: true

Automated review via Coder Agents. Reply to a finding's thread with the reason if the finding is wrong.

Emyrk
Emyrk previously approved these changes Nov 13, 2026

@Emyrk Emyrk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I want to record the intentional policy change here: the 30-minute pending threshold is no longer an absolute limit on the age of queued work. A pending job can remain in the queue without an age cap while an eligible online provisioner is making progress. We are choosing to treat that older work as still valuable, rather than failing it solely because it has waited 30 minutes. That tradeoff is acceptable to me and worth stating explicitly for future readers.

This review text was generated with Coder Agents at @Emyrk's direction.

Document the job reaper's timeouts on the provisioner jobs page. A
pending job fails after 30 minutes unless a matching online provisioner
is making progress, and a job behind a moving queue has no maximum wait.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@dylanhuff-at-coder
dylanhuff-at-coder temporarily deployed to main November 13, 2026 20:22 — with GitHub Actions Inactive
@dylanhuff-at-coder

Copy link
Copy Markdown
Collaborator Author

Good catch, both of these were stale. Updated the provisioner jobs page in bbe80de.

  • The intro no longer says pending jobs hang indefinitely.
  • Added an "Automatic job timeouts" section covering the 30 minute pending timeout, what counts as a provisioner making progress, and that a job behind a moving queue has no max wait.
  • The "stuck in Pending" bullet now mentions the busy queue case and that canceling is the only way to clear it.

I also tweaked the Failed row in the status table since it only mentioned provisioner errors.

@dylanhuff-at-coder
dylanhuff-at-coder merged commit b0ae8ad into main Nov 16, 2026
34 of 35 checks passed
@dylanhuff-at-coder
dylanhuff-at-coder deleted the dylan/reaper-queue-progress-check branch November 16, 2026 13:45
@dylanhuff-at-coder
dylanhuff-at-coder temporarily deployed to main November 16, 2026 13:45 — with GitHub Actions Inactive
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 16, 2026

This branch had an error being deployed

1 failed and 1 inactive deployments
main — bbe80dec Deployed Nov 16, 2026 by dylanhuff-at-coder via cla #257914
docs-preview — bbe80dec Deployed Nov 16, 2026 by dylanhuff-at-coder via cleanup #113
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants