Repository navigation
feat(coderd): protect pending jobs only while the provisioner queue is moving - #32099
Conversation
Preserve pending jobs past the 30-minute timeout while an eligible provisioner is online, including busy provisioners. Add a configurable maximum pending age that defaults to two hours since creation. Filter protected jobs before the candidate limit, recheck eligibility under the job row lock, and reuse availability checks for jobs with matching requirements. Refs: https://linear.app/codercom/issue/CORE-586
…s moving A pending job past PendingJobDuration is now protected only while an eligible online provisioner is making progress: it completed a job within ProvisionerProgressWindow (30m) or is running a job that is not hung. A slow but moving queue is never reaped, and a frozen one is. The progress check is two EXISTS legs so each is a range scan on provisioner_jobs_worker_id_organization_id_completed_at_idx. The locked recheck in reapJob applies the same rule through the new HasProvisionerDaemonMakingProgress query. MaxPendingJobDuration moves from 2h to 24h and is now only a backstop for jobs a moving pool never acquires.
Run the locked recheck against the real database for protected jobs, make the long-build fixture start before the progress window, and cover both progress legs in the candidate exclusion test. Also require provisioner daemon read for HasProvisionerDaemonMakingProgress and reword the pending reap message so it does not depend on the window length.
|
/coder-agents-review Please limit the review to P0, P1, and P2 issues only. |
|
Chat: Review posted | View chat Review historydeep-review v0.13.0 | Round 2 | Last posted: Round 2, 17 findings (6 P3, 5 P4, 6 Nit), APPROVE. Review Finding inventoryFindings
Contested and acknowledgedCRF-1 (P3, provisionerjobs.sql:366) - Reaper-terminated hung job counts as progress
CRF-4 (P4, detector.go:346) - Untested progress-query error in recheck
CRF-5 (Nit, provisionerjobs.sql:414) - Query name lacks By... suffix
CRF-8 (P3, detector.go:341) - Locked recheck duplicates progress rule over unlocked state
CRF-9 (P3, provisionerjobs.sql:374) - Hung running-job guard untested
Round logRound 1Full panel (round 1). Netero first pass: 2 P3, 2 P4, 1 Nit, 2 Note, no P0, so panel ran. Law not run (effective additions 294). Panel: ging-go, knuckle, killua, ryosuke, kurapika, melody, luffy (specialists); pariston, mafuuu (core); bisky, gon, leorio (pool); knov (wildcard). Multi-domain target 9-12; 12 + wildcard, all specialists triggered by Go, SQL, dbauthz, and feature change. Gon's comment-restatement findings (rated P2) downgraded to Nit as project-rule violations with working code and consolidated into CRF-13. CRF-9 linked to CRF-1: the proposed tick-based hung-job test row is order dependent while reaper-terminated hung jobs count as progress (Netero reproduced 4/6 protected). Ging-go, knuckle, killua, kurapika, leorio: no findings; knuckle and killua EXPLAIN runs support the no-migration index claim. 6 P3, 5 P4, 6 Nit posted; 2 dropped; 2 Note (body); 3 OOS. Reviewed against 79d25dd..5512b0c. CRF-1 and CRF-4 author accepted (R2)
CRF-5, CRF-8, CRF-9 panel closure (R2)
Round 2 updateRestructure round (PR rebased standalone against main, About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
…er-progress-check
There was a problem hiding this comment.
This PR protects pending jobs past 30 minutes only while an eligible online provisioner has completed a job in the last 30 minutes or is running a job that is still updated, and raises the absolute cap from 2 hours to 24 hours. Findings: 6 P3, 5 P4, 6 Nit. EXPLAIN runs on seeded PostgreSQL 13 databases showed both progress legs use provisioner_jobs_worker_id_organization_id_completed_at_idx, which supports the no-migration claim.
Out of scope (needs a ticket or explicit acceptance by a human):
coderd/database/migrations/000364_optimize_getprovisionerdaemonswithstatusbyorganization_query.up.sql:3: the indextargets:comment lists onlyGetProvisionerDaemonsWithStatusByOrganization, so someone pruning indexes by that comment would not see that the reaper queries now depend on it.coderd/database/dbauthz/dbauthz.go:5009: theGetProvisionerJobsToBeReapedauthorizer comment restates the authorize call below it.coderd/jobreaper/detector_internal_test.go:54: the-35 * time.Minuteliteral for "past PendingJobDuration" is also on unchanged lines here.
Notes:
- A daemon that comes online after the pool was down has no progress until its first acquisition, so a reaper tick in that window reaps matching jobs older than 30 minutes, where the heartbeat alone protected them before. The window is the time to first acquisition.
- The 24 hour value has no stated basis beyond "backstop". When every eligible daemon runs a build that keeps updating but never finishes, the queue behind it now gets an error after 24 hours instead of 2. That number needs a human decision.
🤖 This review was automatically generated with Coder Agents.
Drop the absolute pending age cap, so a pending job stays protected for as long as an eligible provisioner is online and making progress, and restore main's reap messages and reapJob signature. The progress check, its locked recheck, and the tests that pin them are unchanged.
Use "making progress" consistently, document the stuck-build tradeoff and the caller contract of HasProvisionerDaemonMakingProgress, and rename its progress parameter to match GetProvisionerJobsToBeReaped. Tests derive job ages from PendingJobDuration, and the canceled-before-acquisition case now pins the pending status condition of the protection clause.
|
/coder-agents-review Please limit the review to P0, P1, and P2 issues only. |
There was a problem hiding this comment.
This round reviewed the standalone version that protects pending jobs only while a matching online provisioner is making progress and has no absolute pending cap. As requested, it covers P0, P1, and P2 only, and there are none; 3 P3 and 2 Nit findings were not posted.
The round 1 fixes are in place. The defenses on the query name, the locked recheck, and the hung running-job test were accepted. Two findings were acknowledged without a fix or ticket (a reaper-terminated hung job counting as progress, and the untested progress-query error branch); that acceptance needs to come from a human.
Out of scope (needs a ticket or explicit acceptance by a human):
provisionerd/runner/runner.go:472: builds have no maximum duration, so a build that keeps heartbeating holds its provisioner until someone cancels it; with no pending cap, the queue behind it waits for that cancel too.
🤖 This review was automatically generated with Coder Agents.
Documentation CheckBlocking
These two findings are about Related open PRs
Findings for jobreaper- id: learner-1
persona: learner
class: missing
blocking: true
file: docs/install/operate/provisioners/manage-provisioner-jobs.md
line: 8
quote: "While most jobs complete smoothly, issues with templates, cloud resources, or misconfigured provisioners can cause jobs to fail or hang indefinitely (these are in a `Pending` state)."
evidence:
- coderd/jobreaper/detector.go:26-29
- coderd/jobreaper/detector.go:284-319
rule: null
problem: The page teaches that pending jobs just hang, with no mention that Coder itself automatically fails a pending job after 30 minutes unless a matching, online, "making progress" provisioner exists; a reader troubleshooting a stuck queue would not know some of their pending jobs are being auto-reaped while others, behind a working but backlogged pool, are protected indefinitely and will never auto-clear no matter how old they get.
fix: "Add a note near this paragraph (and in the 'When to cancel provisioner jobs' and 'Troubleshoot provisioner jobs' sections) explaining that Coder's job reaper automatically fails a pending job older than 30 minutes unless an eligible, recently-heartbeating provisioner for that job's organization/type/tags has completed a job in the last 30 minutes or is actively updating a running job within the last 5 minutes; otherwise the job can remain pending indefinitely as long as such a provisioner keeps making progress."
verified: true
- id: coverage-1
persona: coverage
class: missing
blocking: true
file: docs/install/operate/provisioners/manage-provisioner-jobs.md
line: 62
quote: "- It has been stuck in **Pending** for too long.\n This can be due to misconfigured tags or unavailable provisioners."
evidence:
- coderd/jobreaper/detector.go:284-319
- coderd/database/queries/provisionerjobs.sql:328-407
- "coderd/jobreaper/detector.go:388-394 (confirms this is a timing-behavior change only; the stored error text/ErrorCode on a reaped job is unchanged)"
rule: null
problem: Before this PR, every pending provisioner job older than 30 minutes was reaped unconditionally, so a stuck Pending job always auto-resolved within that window even without intervention. This PR makes that reap conditional on provisioner state: a pending job now stays Pending indefinitely, with no absolute age cap, whenever a matching, online provisioner keeps completing other jobs or keeps updating a running job. The bullet lists only "misconfigured tags or unavailable provisioners" and presents cancellation as one option among several remedies, but doesn't mention the new scenario (a correctly tagged, online, busy-but-progressing provisioner pool) or that canceling is now the only way to clear such a job, since the reaper no longer times it out automatically in that case.
fix: "Expand the bullet to cover the new cause and state that cancellation is now the only remedy for it, e.g.: It has been stuck in Pending for too long. This can be due to misconfigured tags, unavailable provisioners, or a busy provisioner pool that keeps making progress on other jobs. Coder only auto-fails a pending job once no matching online provisioner has made progress for a while, so a job behind an otherwise-healthy pool can stay pending indefinitely until you cancel it."
verified: true
- id: accuracy-1
persona: accuracy
class: suggestion
blocking: false
file: coderd/database/queries/provisionerjobs.sql
line: 354
quote: "-- scan on provisioner_jobs_worker_id_organization_id_completed_at_idx."
evidence:
- coderd/database/migrations/000364_optimize_getprovisionerdaemonswithstatusbyorganization_query.up.sql:1
- coderd/database/queries/provisionerjobs.sql:328-407
rule: null
problem: The comment and the PR description both assert the two progress EXISTS legs stay index range scans on provisioner_jobs_worker_id_organization_id_completed_at_idx, but no test pins that plan; a future predicate or column-order change could silently fall back to a sequential scan without failing any test.
fix: Add a planner-pinning regression test (e.g., in coderd/database/queries or a new query_test.go) that runs EXPLAIN on GetProvisionerJobsToBeReaped and HasProvisionerDaemonMakingProgress against a seeded table and asserts the plan uses provisioner_jobs_worker_id_organization_id_completed_at_idx, so a future edit that breaks the index match fails CI instead of only degrading production latency.
verified: trueAutomated review via Coder Agents. Reply to a finding's thread with the reason if the finding is wrong. |
Emyrk
left a comment
There was a problem hiding this comment.
Approving. I want to record the intentional policy change here: the 30-minute pending threshold is no longer an absolute limit on the age of queued work. A pending job can remain in the queue without an age cap while an eligible online provisioner is making progress. We are choosing to treat that older work as still valuable, rather than failing it solely because it has waited 30 minutes. That tradeoff is acceptable to me and worth stating explicitly for future readers.
This review text was generated with Coder Agents at @Emyrk's direction.
Document the job reaper's timeouts on the provisioner jobs page. A pending job fails after 30 minutes unless a matching online provisioner is making progress, and a job behind a moving queue has no maximum wait. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Good catch, both of these were stale. Updated the provisioner jobs page in bbe80de.
I also tweaked the Failed row in the status table since it only mentioned provisioner errors. |
Stops the pending job reaper from failing jobs in a queue that's still moving. This is an alternative to #31469 that came out of Steven's review asking whether 2h is enough, and it builds on George's work there.
A pending job past 30m is now only protected while a matching online provisioner is making progress, meaning it finished a job in the last 30m or is running one that's still being updated. A moving queue never gets reaped and a frozen one still does at 30m like today. There's no absolute cap.
No migration since it uses the existing
provisioner_jobs_worker_id_organization_id_completed_at_idx. One tradeoff is that a stuck build whose provisioner keeps heartbeating counts as progress, so if every matching provisioner is stuck like that, the queue waits until that build is canceled.Still a draft so we can talk it through with Spike and Steven.
Part of CORE-586