Skip to content

Judge a worker by how its requests end, not only by their handlers - #301

Merged
diolektor merged 1 commit into
mainfrom
fix/shutdown-function-fatal-resets-the-error-breaker
Aug 13, 2026
Merged

Judge a worker by how its requests end, not only by their handlers#301
diolektor merged 1 commit into
mainfrom
fix/shutdown-function-fatal-resets-the-error-breaker

Conversation

@diolektor

Copy link
Copy Markdown
Contributor

The consecutive-error breaker retires a worker after three failing requests in a row, and it decides what "failing" means from flags the request fiber's own zend_catch raises. Everything that happens after the request handler returns is outside that arm, and PHP runs it behind guards of its own — so for the whole shutdown window the breaker was not merely blind, it was reading the wrong answer.

A request that came apart there arrived at the check with no flag raised at all. That is the branch that reads a healthy worker and clears the run. So an application fataling in its shutdown function on every request was, to this mechanism, the picture of health, and one alternating between a fatal in its handler and a fatal in its shutdown function held the count at one for the life of the process. The same held for an exception thrown there.

What changed

Four outcomes at the end of a request are now classified the way the same outcome in the handler already was.

Outcome, raised after the handler returned Before After
Fatal in a shutdown function Clears the run Counts
Fatal in a destructor run while the registry is freed Clears the run Counts
Uncaught exception from a shutdown function Clears the run Neutral
max_execution_time expiring in that window Clears the run Neutral — unless the request had already failed, then it still counts

Thirteen lines of code in ext/oxphp_fiber.c. The witnesses were already there and already being read: both CG(unclean_shutdown) sites repair the abandoned frames by hand, which is the definition of a request that came apart, and the exception is already reported a few lines below.

The deadline needs its own handling because max_execution_time is the one cancellation the interrupt handler never sees — the engine calls zend_timeout() ahead of it. Every other cancellation is marked before it unwinds, so it survives whatever swallowed the bailout; this one is recognised by its connection_status bit in the handler arm, which a shutdown-window bailout bypasses. Counting it would rotate the whole pool three requests at a time for an application that defers work to the end of a request and reaches a dependency that has gone slow.

That bit is only read for a request that had not already come apart, and this is the subtle half. It is raised once and stands for the rest of the request, while the bailout flag beside it is lowered by every repair — so on its own it answers "did this request's deadline ever expire", not "did the deadline end this window". Without the guard, a handler fatal followed by a shutdown function outliving the remaining deadline would be filed as a cancellation, erasing a failure already established correctly, and handing any application a way to keep a permanently broken worker in the pool by registering one slow shutdown function.

Tests

Seven sequences appended to the worker-mode breaker profile, each ending in a probe that reads Worker::current()->requestCount() together with oxphp_worker_recycles_by_reason_total{reason="error"} — 29 suite entries, taking the profile from 23 to 52.

Three of the seven are what actually discriminates, and they are the mixed ones. A block of three identical neutral outcomes passes just as well on a build that reads them as successes, because clearing the count does not retire a worker either. Only a neutral outcome dropped into a run of fatals separates "did not count" from "wiped the count". The same asymmetry the profile already had between its cancellation blocks.

Every change in this PR was arrived at by a red test on the build immediately before it, in a chain of five images that differ by one edit each:

image added the block that was red
1 shutdown_retires, shutdown_does_not_clear, shutdown_dtor_retires
2 the failure flag shutdown_timeout_neutral
3 the deadline bit fatal_then_timeout_still_counts
4 the already-failed guard shutdown_throw_does_not_clear
5 the throw flag all green

One trap is worth repeating because the fixture looked right and was not. The destructor case first declared a class directly inside __destruct; PHP rejects that while the file is compiled ("Class declarations may not be nested"), so the fatal happened at include time, inside the handler's own arm, where the outcome is already counted — the block passed on a broken build and only the log showed why. It now reaches E_COMPILE_ERROR at runtime through a require of a helper that declares a class at top level. The point at which the destructor runs was confirmed separately with a non-fataling destructor writing markers: handler returning → shutdown function ran → destructor ran.

Verification

  • breaker 52/52
  • default 242/242, fibers 32/32, timeout 8/8, bailout 2/2
  • worker 54/56 — the two errors are shared/test_channel_fanin_waker and shared/test_channel_async_usevar_stress, known stress flakes; a control run of the same profile on an unmodified image produces the same two
  • cargo fmt --check, cargo clippy --all-targets --no-default-features -D warnings, cargo test --no-default-features — all clean

Notes for review

  • exit() and die() in a shutdown function are untouched: both throw an unwind exit rather than bailing out, so no flag is raised for them and such a request still counts as one that completed. The throw flag explicitly excludes unwind and graceful exit for the same reason.
  • Traditional, framework and SPA serving have no worker to recycle and are unaffected.
  • Two sibling arms at the very end of the request loop still recover from a bailout without marking anything — a __toString that fatals while an exception is reported, and a userland output handler that fatals during the end-of-request flush. Same class, left out of this change deliberately and tracked separately.

Fix:
  - A request whose shutdown function fatals now counts towards the consecutive-error breaker. PHP runs registered shutdown functions — and frees the registry holding them, with the destructors of everything those registrations kept alive — behind guards of its own, so a bailout from either never reached the arm the worker wraps the handler in, and the request arrived at the check with no failure recorded. That is the branch that reads a healthy worker and clears the run, so an application fataling in its shutdown function on every request looked like proof the worker was fine, and one alternating between a fatal in its handler and a fatal in its shutdown function held the count at one for the life of the process. Both sites already repaired the abandoned frames by hand; the flag they repair on is the same witness the count now reads.
  - An exception a shutdown function throws is neutral, in both directions. The loop reports such an exception itself, with a call that asks for no bailout, so no flag was raised for it either and the request cleared the run the same way — while an exception the handler throws has been neutral since the breaker was narrowed. One throwing shutdown function between two fatals therefore kept a worker that failed every other request in the pool indefinitely. exit() and die() land in the same place and are excluded: those are requests that completed.
  - A deadline expiring in that window stays neutral, as it is everywhere else. max_execution_time is the one cancellation the interrupt handler never sees — the engine calls zend_timeout() ahead of it — so unlike a client abort or a drain it is not marked before it unwinds, and the arm that recognises it by its connection_status bit is precisely the arm a shutdown-window bailout bypasses. Without this an application that defers work to the end of a request and reaches a dependency gone slow would rotate the whole pool three requests at a time, which is the harm that made cancellations neutral in the first place.
  - That bit is only read for a request that had not already come apart. It is raised once and stands for the rest of the request, while the bailout flag beside it is lowered by every repair — so on its own it says whether the request's deadline ever expired, not whether the deadline ended this window. A handler fatal followed by a shutdown function outliving the remaining deadline would have been filed as a cancellation, erasing a failure already established correctly and handing any application a way to keep a permanently broken worker in the pool by registering one slow shutdown function.

Docs:
  - Worker-mode recycling no longer states that a fatal in a shutdown function is uncounted, which this makes false. It now says that where a failure was raised does not change how it is read, and which deadline is neutral and when.

Tests:
  - Seven sequences in the worker-mode breaker profile, each ending in a probe that reads requestCount() together with the recycle counter: a fatal in a shutdown function retires the worker, and so does one in a destructor run while the registry is freed; three deadlines expiring in the shutdown window do not, while three handler fatals whose shutdown functions then outlive the deadline still do; and three thrown exceptions do not either.
  - Three of the seven are what actually discriminates, and they are the mixed ones. A block of three identical neutral outcomes passes just as well on a build that reads them as successes, because clearing the count does not retire a worker either — only a neutral outcome dropped into a run of fatals separates "did not count" from "wiped the count".
  - The destructor case reaches its fatal through a require of a helper that declares a class, because a class statement written inside a method is rejected while the file is compiled — which fatals at include time instead, in the handler, where the outcome is already counted and the test would have passed on a broken build.

29 integration tests.
@diolektor
diolektor merged commit 1e81d49 into main Aug 13, 2026
7 checks passed
@diolektor
diolektor deleted the fix/shutdown-function-fatal-resets-the-error-breaker branch August 13, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant