Repository navigation
Conversation
Preserve pending jobs past the 30-minute timeout while an eligible provisioner is online, including busy provisioners. Add a configurable maximum pending age that defaults to two hours since creation. Filter protected jobs before the candidate limit, recheck eligibility under the job row lock, and reuse availability checks for jobs with matching requirements. Refs: https://linear.app/codercom/issue/CORE-586
|
/coder-agents-review Please limit the review to P0, P1, and P2 issues only. |
|
Chat: Review posted | View chat Review history
deep-review v0.13.0 | Round 1 | Last posted: Round 1, 19 findings (8 P3, 5 P4, 6 Nit), COMMENT. Review Finding inventoryFindings
Contested and acknowledgedRound logRound 1Full panel (round 1). Netero: 1 P4, 1 Note, no P0, panel proceeds. Law not run (effective additions 857 <= 1000). Panel: ging-go, knuckle, killua, kurapika, luffy, ryosuke (specialists); pariston, mafuuu (core); bisky, hisoka, gon, leorio (pool); kite (wildcard). Multi-domain (Go + SQL + RBAC grant), 12 + wildcard. Cross-check decisions:
About deep-reviewCRF = Coder Review Finding (P0-P4, Nit, Note)
|
There was a problem hiding this comment.
This PR keeps pending provisioner jobs from being reaped at 30 minutes while an eligible provisioner is online, adds --job-reaper-max-pending-age (default 2h) as an absolute limit, and grants the reaper provisioner_daemon:read. Findings: 8 P3, 5 P4, 6 Nit.
Out of scope (needs a ticket or explicit acceptance by a human):
coderd/jobreaper/detector.go:229:runlabels every unstarted non-pending candidateHungwith a 5-minute threshold, so a job canceled before acquisition and idle for 30 minutes fails with "detected as hung for 5 minutes"; the newCanceledBeforeAcquisitiontest pins that label.coderd/database/queries/provisionerjobs.sql:385: the hung-job branch matches no index, so each reaper tick likely scans all ofprovisioner_jobs(not verified with EXPLAIN).provisioner/terraform/serve.go:135: the provisioner runtime imports the server-sidejobreaperpackage for one constant, so every dependency added tojobreaperalso lands inprovisioner/terraform.coderd/jobreaper/detector.go:310: the hung-job "updated recently" recheck has no test coverage, so a regression there would not be caught.coderd/jobreaper/detector.go:39: theMaxJobsPerRundoc says hung jobs, but the limit also applies to pending jobs.coderd/jobreaper/detector.go:101: theDetectordoc says it detects hung jobs, but it also reaps pending jobs.coderd/jobreaper/detector.go:360: the insert-logs error formatsjob.JobStatuswhile its sibling errors format the reap type, so errors for the same job name it differently.
Notes:
coderd/jobreaper/detector.go:128:Newaccepts anymaxPendingAge; a zero value would reap every pending job on each tick. All current callers pass a validated or fixed value.coderd/jobreaper/detector.go:293: the recheck reads daemons through a dbauthz post-filter, so a missingprovisioner_daemon:readgrant would reap protected jobs instead of erroring. The explicit check added toGetProvisionerJobsToBeReapedcurrently fails the run first.coderd/jobreaper/detector.go:285: when the backlog exceeds the maximum age, the reaped jobs are the oldest ones, whichAcquireProvisionerJobwould run next. The old 30-minute rule behaved the same way.coderd/jobreaper/detector_internal_test.go:51: the recheck's online branch is covered only by a gomock that returns a daemon unconditionally.- The PR description does not mention that the job
Errortext changes; alerts or log searches matching "detected as pending for" will not match max-age reaps.
🤖 This review was automatically generated with Coder Agents.
spikecurtis
left a comment
There was a problem hiding this comment.
Some specific comments inline, but the tests here are definitely slop and need to be looked at. We're adding dozens of new tests cases for what, on face value, is small change in behavior.
| Annotations: serpent.Annotations{}.Mark(annotationFormatDuration, "true"), | ||
| }, | ||
| { | ||
| Name: "Job Reaper Maximum Pending Age", |
There was a problem hiding this comment.
We have so many configurable options, we really need to set a high bar for adding new knobs for operators to know about and tweak.
So, do we really need this to be adjustable? Under what circumstances would an operator legitimately need to tune it?
Keep in mind that we can always add a knob later, but once we have it, it's very difficult to take away.
There was a problem hiding this comment.
Agreed, I couldn't find a case for tuning it either. Made it a constant, MaxPendingJobDuration (2h).
There was a problem hiding this comment.
Under what circumstances would an operator legitimately need to tune it?
For example a deployment with low provisioner capacity where builds can legitimately remain queued for more than two hours while eligible provisioners are online and busy. The underlying ticket already describes a similar use case (but with 30 min instead of 2 hours).
We could remove the unconditional reaping, but then a provisioner that keeps heartbeating but stops acquiring jobs could leave builds queued indefinitely.
| // maxPendingAge must be greater than [PendingJobDuration]. | ||
| // provisionerStaleInterval is the maximum time since a provisioner's last | ||
| // heartbeat for it to be considered online. | ||
| func New(ctx context.Context, db database.Store, pub pubsub.Pubsub, log slog.Logger, tick <-chan time.Time, maxPendingAge, provisionerStaleInterval time.Duration) (*Detector, error) { |
There was a problem hiding this comment.
provisionerStateInterval is always set to a constant value, even in tests. Why does it need to be an argument of this function?
There was a problem hiding this comment.
Looks like it was passed in because jobreaper can't import provisionerdserver without an import cycle. Made it a constant instead. It's 5m, longer than the UI's 90s, so one missed heartbeat doesn't expose old jobs to reaping.
| FROM provisioner_jobs pj | ||
| WHERE pj.started_at IS NULL | ||
| AND pj.completed_at IS NULL | ||
| AND pj.job_status = 'pending'::provisioner_job_status |
There was a problem hiding this comment.
There is a relatively subtle change here that I don't understand. In the existing job reaper, we select jobs that are not started and not completed. In the new version, we select jobs that are not started, not completed AND have a pending status.
First of all, pending implies not started and not completed. The job status is a derived column.
Secondly, there are cases where the job is not started, not completed, but NOT pending, which you can see if you look at the definition of job_status. In particular, jobs that have canceled_at not null could technically have started_at be null and be subject to the old job reaper query, but not this one.
Is that change intentional? If so, I think it's sufficient to simply to just select pending. If not, then we should be careful about making it.
There was a problem hiding this comment.
Doesn't look intentional. I restored main's predicate as-is and only added the protection clause, so canceled jobs behave like main. CanceledBeforeAcquisition covers it.
|
Hey, I'm picking this up while George is on PTO. I pushed a commit addressing @spikecurtis feedback. I'm still getting up to speed, so let me know if I missed any context.
Agreed, I trimmed the tests down a lot. The PR is 8 files now instead of 23. A few notes:
Also I reviewed this, made changes and would approve, but I have edited it enough now that it warrants a third person approving before merging |
Emyrk
left a comment
There was a problem hiding this comment.
Do we know the queue delays seen in CORE-586, and is two hours enough to cover them?
I understand this PR now allows up to 2 hours for a provisioner to work through the queue. It would be great to understand if that is sufficient? Is a 2 hour queue considered "ok"?
--
A lot of this feels kind of arbitrary, and I fear this patch will just come back with the message saying 2 hours instead of 30 minutes.
Coder: Build has been detected as pending for 2 hours and has been terminated by the reaper.
Do we really need a hard backstop where we kill the job? Maybe send a notification to the admin about builds that are hanging a really long time (ideally one notification about all hung jobs with the same org and labels). |
| -- function, which Postgres cannot inline, instead of jobs x daemons. | ||
| SELECT DISTINCT organization_id, provisioner, tags | ||
| FROM provisioner_jobs | ||
| WHERE job_status = 'pending'::provisioner_job_status |
There was a problem hiding this comment.
We still seem to be mixing and matching in this query whether we are looking at job_status = pending, or at started_at is NULL and completed_at is NULL. These are usually equivalent, but not in certain edge cases, so it should be one or the other.
|
|
||
| // A healthy provisioner pool can have a long queue. Preserve pending jobs | ||
| // while an eligible provisioner is online, subject to the absolute age limit. | ||
| if reapType == Pending { |
There was a problem hiding this comment.
I think this is overkill. It's one thing to make sure that the job hasn't been started in the time between us determining its elegibility for reaping and the actual moment we reap it. That takes one query to the job table itself.
Here we're essentially recreating a bunch of DB joins in Go code using a long transaction.
Yes, technically, between the time we queried this job and the time we reap it a new provisioner could come online that unblocks the queue, but if it's been sitting for at least 30 minutes, the chances are astronomically low that is going to happen in those exact milliseconds between the elegibility query and this one. Even if we magically hit that tiny window, we're just back to the old behavior where we reaped unconditionally after 30 minutes.
Holding open a long transaction blocks other DB work at scale, and so I just don't think this is worth it.
…s moving (#32099) Stops the pending job reaper from failing jobs in a queue that's still moving. This is an alternative to #31469 that came out of Steven's [review](#31469 (review)) asking whether 2h is enough, and it builds on George's work there. A pending job past 30m is now only protected while a matching online provisioner is making progress, meaning it finished a job in the last 30m or is running one that's still being updated. A moving queue never gets reaped and a frozen one still does at 30m like today. There's no absolute cap. No migration since it uses the existing `provisioner_jobs_worker_id_organization_id_completed_at_idx`. One tradeoff is that a stuck build whose provisioner keeps heartbeating counts as progress, so if every matching provisioner is stuck like that, the queue waits until that build is canceled. Still a draft so we can talk it through with Spike and Steven. Part of CORE-586 --------- Co-authored-by: George Katsitadze <george.katsitadze@gmail.com> Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
|
Since #32099 was merged I'm going to close this one. The description say it's an alternative fix. Plz reopen if I'm misinterpreting cc @dylanhuff-at-coder |
Synchronized autostops and autostarts can queue more builds than provisioners can work through in 30 minutes. The reaper fails any pending job that hasn't been updated in 30 minutes, whether or not a provisioner could still pick it up, so valid builds get reaped from a queue that's backed up but healthy.
Fix
The reaper now skips pending jobs while an eligible provisioner is online, meaning same organization, provisioner type and matching tags, seen within the last 5 minutes. Busy provisioners count. Jobs with no eligible online provisioner are still reaped after 30 minutes like before. Every pending job becomes reapable after 2 hours (
jobreaper.MaxPendingJobDuration), so jobs stuck behind a provisioner that's online but not making progress still get cleaned up.The 5 minute staleness is intentionally longer than the 90 second threshold the UI uses, so one missed heartbeat during a burst doesn't expose long queued jobs to reaping.
Protected jobs are filtered out before the candidate limit and rechecked under the row lock before reaping. Daemon eligibility is checked once per distinct requirement set (organization, provisioner, tags) instead of once per job. In a local benchmark with 2M completed jobs and 10k pending, the candidate query takes about 7ms. The reaper now needs
provisioner_daemon:read.Running-job cleanup is unchanged. This needs to ship in a patch release, so there are no migrations.
Jitter on scheduled starts and stops was considered, but it can't guarantee waits under 30 minutes and leaves the reaping behavior as is. It could still be a follow-up to smooth out the herd.
Refs: https://linear.app/codercom/issue/CORE-586
Note:
@dylanhuff-at-coder is picking this up from @geokat while he is OOO