Skip to content

fix: avoid premature reaping of pending provisioner jobs - #31469

Closed
geokat wants to merge 3 commits into
mainfrom
george/core-586/fix/protect-pending-provisioner-jobs
Closed

geokat wants to merge 3 commits into
mainfrom
george/core-586/fix/protect-pending-provisioner-jobs

Conversation

@geokat

@geokat geokat commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

Synchronized autostops and autostarts can queue more builds than provisioners can work through in 30 minutes. The reaper fails any pending job that hasn't been updated in 30 minutes, whether or not a provisioner could still pick it up, so valid builds get reaped from a queue that's backed up but healthy.

Fix

The reaper now skips pending jobs while an eligible provisioner is online, meaning same organization, provisioner type and matching tags, seen within the last 5 minutes. Busy provisioners count. Jobs with no eligible online provisioner are still reaped after 30 minutes like before. Every pending job becomes reapable after 2 hours (jobreaper.MaxPendingJobDuration), so jobs stuck behind a provisioner that's online but not making progress still get cleaned up.

The 5 minute staleness is intentionally longer than the 90 second threshold the UI uses, so one missed heartbeat during a burst doesn't expose long queued jobs to reaping.

Protected jobs are filtered out before the candidate limit and rechecked under the row lock before reaping. Daemon eligibility is checked once per distinct requirement set (organization, provisioner, tags) instead of once per job. In a local benchmark with 2M completed jobs and 10k pending, the candidate query takes about 7ms. The reaper now needs provisioner_daemon:read.

Running-job cleanup is unchanged. This needs to ship in a patch release, so there are no migrations.

Jitter on scheduled starts and stops was considered, but it can't guarantee waits under 30 minutes and leaves the reaping behavior as is. It could still be a follow-up to smooth out the herd.

Refs: https://linear.app/codercom/issue/CORE-586

Note:

@dylanhuff-at-coder is picking this up from @geokat while he is OOO

Preserve pending jobs past the 30-minute timeout while an eligible
provisioner is online, including busy provisioners. Add a configurable
maximum pending age that defaults to two hours since creation.

Filter protected jobs before the candidate limit, recheck eligibility
under the job row lock, and reuse availability checks for jobs with
matching requirements.

Refs:
https://linear.app/codercom/issue/CORE-586
@linear-code

linear-code Bot commented Oct 28, 2026

Copy link
Copy Markdown

CORE-586

@geokat

geokat commented Oct 28, 2026

Copy link
Copy Markdown
Collaborator Author

/coder-agents-review Please limit the review to P0, P1, and P2 issues only.

@coder-agents-review

coder-agents-review Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-10-28 23:31 UTC by @geokat

Review history
  • R1 (2026-10-28): 13 reviewers, 6 Nit, 8 P3, 5 P4, COMMENT. Review

deep-review v0.13.0 | Round 1 | edd2e07..fb65a84

Last posted: Round 1, 19 findings (8 P3, 5 P4, 6 Nit), COMMENT. Review

Finding inventory

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P4 Open codersdk/deployment.go:2396 Validator and description hardcode copy of PendingJobDuration R1 Netero Yes
CRF-2 Note Note (body) coderd/jobreaper/detector.go:128 New accepts unvalidated maxPendingAge; zero would reap every pending job R1 Netero No
CRF-3 P3 Open coderd/database/queries/provisionerjobs.sql:362 LEFT JOIN planned as nested loop; 243 ms vs 10 ms on 2M-row history R1 Killua Yes
CRF-4 P3 Open coderd/database/queries/provisionerjobs.sql:329 Simpler alternative (2h timeout alone) not evaluated; availability check keeps 90s-gap failure mode R1 Pariston P3, Luffy Note Yes
CRF-5 P3 Open coderd/jobreaper/detector.go:18 Import of provisionerdserver doubles provisioner/terraform dependency graph R1 Ryosuke Yes
CRF-6 P3 Open codersdk/deployment.go:2389 Visible ungrouped option makes top-level YAML/JSON key a contract R1 Ryosuke P3, Kite Nit Yes
CRF-7 P4 Open coderd/jobreaper/detector.go:293 Go recheck is a second eligibility definition with no keep-in-sync note R1 Ryosuke P4, Gon P2 (detector.go:277) Yes
CRF-8 P3 Open coderd/jobreaper/pending_test.go:97 Protected-row tests pass when CTE protection regresses (recheck masks it) R1 Bisky Yes
CRF-9 Note Note (body) coderd/jobreaper/detector_internal_test.go:51 Recheck online branch proven only against gomock R1 Bisky No
CRF-10 Note Note (body) coderd/jobreaper/detector.go:293 Recheck uses post-filter; authz denial would reap instead of error, guarded by explicit dbauthz check R1 Kurapika, Kite No
CRF-11 Note Note (body) coderd/jobreaper/detector.go:285 Backlog beyond max age reaps the jobs at the front of the queue R1 Luffy No
CRF-12 P3 Open coderd/jobreaper/detector.go:58 Pending reap message can state wrong wait time and gives no next step R1 Leorio Yes
CRF-13 P3 Open coderd/jobreaper/detector.go:60 Max-age reap message does not name the setting R1 Leorio Yes
CRF-14 P3 Open codersdk/deployment.go:2391 Option description leaves eligible/online and the two clocks undefined R1 Leorio Yes
CRF-15 Nit Open coderd/database/queries/provisionerjobs.sql:394 Unchanged comments rewrapped R1 Leorio Nit, Gon P2 Yes
CRF-16 Nit Open coderd/database/queries/provisionerjobs.sql:330 SQL comments at 330, 337, 344, 368, 377, 380 restate the SQL R1 Gon P2 Yes
CRF-17 Nit Open coderd/jobreaper/pending_test.go:183 GIVEN/WHEN/THEN and other test comments restate code (pending_test.go, detector_internal_test.go:99, pending_bench_test.go:35,49) R1 Gon P2 Yes
CRF-18 P4 Open coderd/jobreaper/pending_test.go:282 "120 minutes" literal coupled to unnamed 2h test value R1 Gon P3 Yes
CRF-19 P4 Open coderd/jobreaper/detector.go:275 reapType and jobToReap.Type disagree in logs and errors for max-age reaps R1 Gon Yes
CRF-20 P4 Open coderd/jobreaper/detector.go:78 PendingMaxAge text never contains its value; requireTerminatedJob cannot check it R1 Gon Yes
CRF-21 Nit Open coderd/jobreaper/detector.go:44 JobLogMessages doc says values "are written" R1 Gon Yes
CRF-22 Nit Open coderd/jobreaper/pending_test.go:67 Two tests use different names for the same mismatch cases R1 Gon Yes
CRF-23 Nit Open coderd/jobreaper/pending_test.go:119 Bare duration literals instead of values derived from the thresholds R1 Gon Yes
CRF-24 Note Dropped by orchestrator (praise, asks for no change) coderd/database/dbauthz/dbauthz.go:5009 Praise for explicit ProvisionerDaemon read check R1 Kite No
CRF-25 Note Dropped by orchestrator (praise, asks for no change) coderd/jobreaper/pending_test.go:64 Praise for availability table coverage R1 Kite No
CRF-26 Note Note (body) commit fb65a84 Description does not mention the user-visible job Error text change R1 Leorio No
CRF-27 OOS Out of scope (body) coderd/jobreaper/detector.go:229 Unstarted canceled jobs labeled Hung for 5 minutes R1 Bisky, Gon, Leorio, Kite No
CRF-28 OOS Out of scope (body) coderd/database/queries/provisionerjobs.sql:385 Hung branch matches no index R1 Knuckle No
CRF-29 OOS Out of scope (body) provisioner/terraform/serve.go:135 Provisioner runtime imports server-side jobreaper R1 Ryosuke No
CRF-30 OOS Out of scope (body) coderd/jobreaper/detector.go:310 Hung recheck branch has no coverage R1 Ryosuke No
CRF-31 OOS Out of scope (body) coderd/jobreaper/detector.go:39 MaxJobsPerRun doc says hung jobs only R1 Gon No
CRF-32 OOS Out of scope (body) coderd/jobreaper/detector.go:101 Detector doc says hung jobs only R1 Gon No
CRF-33 OOS Out of scope (body) coderd/jobreaper/detector.go:360 Insert-logs error names status where siblings name reap type R1 Gon No

Contested and acknowledged

Round log

Round 1

Full panel (round 1). Netero: 1 P4, 1 Note, no P0, panel proceeds. Law not run (effective additions 857 <= 1000). Panel: ging-go, knuckle, killua, kurapika, luffy, ryosuke (specialists); pariston, mafuuu (core); bisky, hisoka, gon, leorio (pool); kite (wildcard). Multi-domain (Go + SQL + RBAC grant), 12 + wildcard.

Cross-check decisions:

  • CRF-6: Ryosuke P3 and Kite Nit converge; higher wins.
  • CRF-4: Pariston P3 and Luffy Note converge on the same alternative; higher wins.
  • CRF-7: Ryosuke P4 (detector.go:293) and Gon P2 (detector.go:277 comment) merged; same concern, two fixes. Keep-argument for Gon P2: drift between the two definitions silently changes reaping. Settled P4 because Ryosuke traced the consequence and the comment is the minor half.
  • CRF-16, CRF-17: Gon raised 21 separate P2 comment-restatement findings. Keep-argument: AGENTS.md requires substantive comments and 25 of 35 comments restate code. Downgraded to Nit because the code works and the violation is a project style rule; grouped into two findings (SQL file, test files) because they are one issue repeated.
  • CRF-15: Gon P2 at 388 and 394 merged with Leorio Nit on the same rewrap. Same Nit reasoning.
  • CRF-18: Gon P3. Keep-argument: hidden cross-file coupling. Downgraded to P4 because the failure mode is a loud test failure in the same package.
  • CRF-5: verified go list -deps ./provisioner/terraform | wc -l = 2228 at head. All importers of provisioner/terraform are fat-binary packages, so the effect is compile graph and cycle risk; kept P3.
  • Hisoka, Mafuuu, Ging-Go: no findings. Knuckle: no in-scope findings.

Reviewed against edd2e07..fb65a84.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

🤖 Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR keeps pending provisioner jobs from being reaped at 30 minutes while an eligible provisioner is online, adds --job-reaper-max-pending-age (default 2h) as an absolute limit, and grants the reaper provisioner_daemon:read. Findings: 8 P3, 5 P4, 6 Nit.

Out of scope (needs a ticket or explicit acceptance by a human):

  • coderd/jobreaper/detector.go:229: run labels every unstarted non-pending candidate Hung with a 5-minute threshold, so a job canceled before acquisition and idle for 30 minutes fails with "detected as hung for 5 minutes"; the new CanceledBeforeAcquisition test pins that label.
  • coderd/database/queries/provisionerjobs.sql:385: the hung-job branch matches no index, so each reaper tick likely scans all of provisioner_jobs (not verified with EXPLAIN).
  • provisioner/terraform/serve.go:135: the provisioner runtime imports the server-side jobreaper package for one constant, so every dependency added to jobreaper also lands in provisioner/terraform.
  • coderd/jobreaper/detector.go:310: the hung-job "updated recently" recheck has no test coverage, so a regression there would not be caught.
  • coderd/jobreaper/detector.go:39: the MaxJobsPerRun doc says hung jobs, but the limit also applies to pending jobs.
  • coderd/jobreaper/detector.go:101: the Detector doc says it detects hung jobs, but it also reaps pending jobs.
  • coderd/jobreaper/detector.go:360: the insert-logs error formats job.JobStatus while its sibling errors format the reap type, so errors for the same job name it differently.

Notes:

  • coderd/jobreaper/detector.go:128: New accepts any maxPendingAge; a zero value would reap every pending job on each tick. All current callers pass a validated or fixed value.
  • coderd/jobreaper/detector.go:293: the recheck reads daemons through a dbauthz post-filter, so a missing provisioner_daemon:read grant would reap protected jobs instead of erroring. The explicit check added to GetProvisionerJobsToBeReaped currently fails the run first.
  • coderd/jobreaper/detector.go:285: when the backlog exceeds the maximum age, the reaped jobs are the oldest ones, which AcquireProvisionerJob would run next. The old 30-minute rule behaved the same way.
  • coderd/jobreaper/detector_internal_test.go:51: the recheck's online branch is covered only by a gomock that returns a daemon unconditionally.
  • The PR description does not mention that the job Error text changes; alerts or log searches matching "detected as pending for" will not match max-age reaps.

🤖 This review was automatically generated with Coder Agents.

Comment thread coderd/jobreaper/detector.go Resolved
Comment thread codersdk/deployment.go Resolved
Comment thread coderd/jobreaper/detector.go Resolved
Comment thread coderd/jobreaper/pending_test.go Resolved
Comment thread coderd/jobreaper/detector.go Resolved
Comment thread coderd/jobreaper/pending_test.go Resolved
Comment thread coderd/jobreaper/pending_test.go Resolved
@geokat
geokat marked this pull request as ready for review October 29, 2026 06:47

@spikecurtis spikecurtis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some specific comments inline, but the tests here are definitely slop and need to be looked at. We're adding dozens of new tests cases for what, on face value, is small change in behavior.

Comment thread cli/server.go Resolved
Comment thread codersdk/deployment.go Outdated
Annotations: serpent.Annotations{}.Mark(annotationFormatDuration, "true"),
},
{
Name: "Job Reaper Maximum Pending Age",

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have so many configurable options, we really need to set a high bar for adding new knobs for operators to know about and tweak.

So, do we really need this to be adjustable? Under what circumstances would an operator legitimately need to tune it?

Keep in mind that we can always add a knob later, but once we have it, it's very difficult to take away.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I couldn't find a case for tuning it either. Made it a constant, MaxPendingJobDuration (2h).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Under what circumstances would an operator legitimately need to tune it?

For example a deployment with low provisioner capacity where builds can legitimately remain queued for more than two hours while eligible provisioners are online and busy. The underlying ticket already describes a similar use case (but with 30 min instead of 2 hours).

We could remove the unconditional reaping, but then a provisioner that keeps heartbeating but stops acquiring jobs could leave builds queued indefinitely.

Comment thread coderd/jobreaper/detector.go Outdated
// maxPendingAge must be greater than [PendingJobDuration].
// provisionerStaleInterval is the maximum time since a provisioner's last
// heartbeat for it to be considered online.
func New(ctx context.Context, db database.Store, pub pubsub.Pubsub, log slog.Logger, tick <-chan time.Time, maxPendingAge, provisionerStaleInterval time.Duration) (*Detector, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

provisionerStateInterval is always set to a constant value, even in tests. Why does it need to be an argument of this function?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like it was passed in because jobreaper can't import provisionerdserver without an import cycle. Made it a constant instead. It's 5m, longer than the UI's 90s, so one missed heartbeat doesn't expose old jobs to reaping.

FROM provisioner_jobs pj
WHERE pj.started_at IS NULL
AND pj.completed_at IS NULL
AND pj.job_status = 'pending'::provisioner_job_status

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a relatively subtle change here that I don't understand. In the existing job reaper, we select jobs that are not started and not completed. In the new version, we select jobs that are not started, not completed AND have a pending status.

First of all, pending implies not started and not completed. The job status is a derived column.

Secondly, there are cases where the job is not started, not completed, but NOT pending, which you can see if you look at the definition of job_status. In particular, jobs that have canceled_at not null could technically have started_at be null and be subject to the old job reaper query, but not this one.

Is that change intentional? If so, I think it's sufficient to simply to just select pending. If not, then we should be careful about making it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't look intentional. I restored main's predicate as-is and only added the protection clause, so canceled jobs behave like main. CanceledBeforeAcquisition covers it.

Comment thread coderd/jobreaper/pending_test.go Resolved
Comment thread coderd/jobreaper/pending_test.go Resolved
Comment thread coderd/jobreaper/pending_test.go Resolved
Comment thread coderd/jobreaper/pending_test.go Resolved
@dylanhuff-at-coder

dylanhuff-at-coder commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Hey, I'm picking this up while George is on PTO. I pushed a commit addressing @spikecurtis feedback. I'm still getting up to speed, so let me know if I missed any context.

We're adding dozens of new tests cases for what, on face value, is small change in behavior.

Agreed, I trimmed the tests down a lot. The PR is 8 files now instead of 23.

A few notes:

  • The 2h max pending age and 5m provisioner staleness (vs 90s in the UI) are judgment calls, open to other values
  • Past 2h, jobs over the cap get reaped at random while the queue drains oldest first, so a really long backlog can still lose jobs. Seems fine for a backstop
  • A provisioner that's wedged but still heartbeating now holds its jobs for 2h instead of 30m
  • This fixes reaping, not the herd itself. Jitter can be a follow-up, and CORE-586 should probably be retitled
  • Jobs canceled before pickup still get a "hung for 5 minutes" message. That's the same on main and out of scope here
  • I'll update the PR description to match

Also I reviewed this, made changes and would approve, but I have edited it enough now that it warrants a third person approving before merging

@Emyrk Emyrk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we know the queue delays seen in CORE-586, and is two hours enough to cover them?

I understand this PR now allows up to 2 hours for a provisioner to work through the queue. It would be great to understand if that is sufficient? Is a 2 hour queue considered "ok"?

--

A lot of this feels kind of arbitrary, and I fear this patch will just come back with the message saying 2 hours instead of 30 minutes.

Coder: Build has been detected as pending for 2 hours and has been terminated by the reaper.

@spikecurtis

Copy link
Copy Markdown
Collaborator

Do we know the queue delays seen in CORE-586, and is two hours enough to cover them?

I understand this PR now allows up to 2 hours for a provisioner to work through the queue. It would be great to understand if that is sufficient? Is a 2 hour queue considered "ok"?

--

A lot of this feels kind of arbitrary, and I fear this patch will just come back with the message saying 2 hours instead of 30 minutes.

Coder: Build has been detected as pending for 2 hours and has been terminated by the reaper.

Do we really need a hard backstop where we kill the job? Maybe send a notification to the admin about builds that are hanging a really long time (ideally one notification about all hung jobs with the same org and labels).

-- function, which Postgres cannot inline, instead of jobs x daemons.
SELECT DISTINCT organization_id, provisioner, tags
FROM provisioner_jobs
WHERE job_status = 'pending'::provisioner_job_status

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We still seem to be mixing and matching in this query whether we are looking at job_status = pending, or at started_at is NULL and completed_at is NULL. These are usually equivalent, but not in certain edge cases, so it should be one or the other.


// A healthy provisioner pool can have a long queue. Preserve pending jobs
// while an eligible provisioner is online, subject to the absolute age limit.
if reapType == Pending {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is overkill. It's one thing to make sure that the job hasn't been started in the time between us determining its elegibility for reaping and the actual moment we reap it. That takes one query to the job table itself.

Here we're essentially recreating a bunch of DB joins in Go code using a long transaction.

Yes, technically, between the time we queried this job and the time we reap it a new provisioner could come online that unblocks the queue, but if it's been sitting for at least 30 minutes, the chances are astronomically low that is going to happen in those exact milliseconds between the elegibility query and this one. Even if we magically hit that tiny window, we're just back to the old behavior where we reaped unconditionally after 30 minutes.

Holding open a long transaction blocks other DB work at scale, and so I just don't think this is worth it.

dylanhuff-at-coder added a commit that referenced this pull request Nov 16, 2026
…s moving (#32099)

Stops the pending job reaper from failing jobs in a queue that's still
moving. This is an alternative to #31469 that came out of Steven's
[review](#31469 (review))
asking whether 2h is enough, and it builds on George's work there.

A pending job past 30m is now only protected while a matching online
provisioner is making progress, meaning it finished a job in the last
30m or is running one that's still being updated. A moving queue never
gets reaped and a frozen one still does at 30m like today. There's no
absolute cap.

No migration since it uses the existing
`provisioner_jobs_worker_id_organization_id_completed_at_idx`. One
tradeoff is that a stuck build whose provisioner keeps heartbeating
counts as progress, so if every matching provisioner is stuck like that,
the queue waits until that build is canceled.

Still a draft so we can talk it through with Spike and Steven.

Part of CORE-586

---------

Co-authored-by: George Katsitadze <george.katsitadze@gmail.com>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
@spikecurtis

Copy link
Copy Markdown
Collaborator

Since #32099 was merged I'm going to close this one. The description say it's an alternative fix. Plz reopen if I'm misinterpreting cc @dylanhuff-at-coder

@spikecurtis
spikecurtis temporarily deployed to main November 17, 2026 12:35 — with GitHub Actions Inactive
@github-actions github-actions Bot locked and limited conversation to collaborators Nov 17, 2026

This branch had an error being deployed

1 failed and 1 inactive deployments
docs-preview — 79d25dd5 Deployed Nov 17, 2026 by spikecurtis via cleanup #163
main — 79d25dd5 Deployed Nov 17, 2026 by spikecurtis via cla #258626
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants