Repository navigation
[Bug] Pipeline worker no-progress can leave jobs wedged indefinitely #4342
Description
Summary
A background pipeline worker can stop advancing a job while the server remains available. The job last_processed_at then remains stale, there is no useful per-worker heartbeat to distinguish a worker stall from normal queueing, and recovery may depend on a server process restart. The underlying stall trigger is not yet established.
Expected behavior
- Expose worker/task heartbeat and pending-work/progress metrics so a stalled pipeline is observable.
- Ensure per-item exceptions record an error and advance retry/progress metadata before the item is retried or backed off.
- Do not silently stop processing a job after a transient exception.
Please add regression tests for a failing item followed by successful retry and for stale-worker detection. Related historical reports include #825 and #3886, but this issue focuses on persistent worker no-progress and observability.
Activity
github-actions commentedon Nov 6, 2026
This issue is stale because it has been open for 30 days with no activity.
Outvoker commentedon Nov 12, 2026
Reproduced on 0.22.5, which already includes #4354 and #4374, so neither of those fixes it.
submittedwithlast_processed_at == submitted_at, the oldest for ~20 h. Other jobs submitted in the same window were provisioned normally.Updating lock_expires_at for items: [<those ids>]every ~20 s, but neverProcessing jobs itemfor them.idle in transaction/ClientReadfor 2.5–6.8 h. The last statement in each wasSELECT fleets..., with no locks beyondAccessShareLockand nothing blocked behind them. So ajobs_submittedworker is hanging on an await after loading fleets, not on the DB.process()inpipeline_tasks/base.pyhas no timeout, and the heartbeat keeps renewing the lock, so one hung await pins the job forever.