Judge a worker by how its requests end, not only by their handlers - #301
Merged
diolektor merged 1 commit intoAug 13, 2026
Merged
Conversation
Fix: - A request whose shutdown function fatals now counts towards the consecutive-error breaker. PHP runs registered shutdown functions — and frees the registry holding them, with the destructors of everything those registrations kept alive — behind guards of its own, so a bailout from either never reached the arm the worker wraps the handler in, and the request arrived at the check with no failure recorded. That is the branch that reads a healthy worker and clears the run, so an application fataling in its shutdown function on every request looked like proof the worker was fine, and one alternating between a fatal in its handler and a fatal in its shutdown function held the count at one for the life of the process. Both sites already repaired the abandoned frames by hand; the flag they repair on is the same witness the count now reads. - An exception a shutdown function throws is neutral, in both directions. The loop reports such an exception itself, with a call that asks for no bailout, so no flag was raised for it either and the request cleared the run the same way — while an exception the handler throws has been neutral since the breaker was narrowed. One throwing shutdown function between two fatals therefore kept a worker that failed every other request in the pool indefinitely. exit() and die() land in the same place and are excluded: those are requests that completed. - A deadline expiring in that window stays neutral, as it is everywhere else. max_execution_time is the one cancellation the interrupt handler never sees — the engine calls zend_timeout() ahead of it — so unlike a client abort or a drain it is not marked before it unwinds, and the arm that recognises it by its connection_status bit is precisely the arm a shutdown-window bailout bypasses. Without this an application that defers work to the end of a request and reaches a dependency gone slow would rotate the whole pool three requests at a time, which is the harm that made cancellations neutral in the first place. - That bit is only read for a request that had not already come apart. It is raised once and stands for the rest of the request, while the bailout flag beside it is lowered by every repair — so on its own it says whether the request's deadline ever expired, not whether the deadline ended this window. A handler fatal followed by a shutdown function outliving the remaining deadline would have been filed as a cancellation, erasing a failure already established correctly and handing any application a way to keep a permanently broken worker in the pool by registering one slow shutdown function. Docs: - Worker-mode recycling no longer states that a fatal in a shutdown function is uncounted, which this makes false. It now says that where a failure was raised does not change how it is read, and which deadline is neutral and when. Tests: - Seven sequences in the worker-mode breaker profile, each ending in a probe that reads requestCount() together with the recycle counter: a fatal in a shutdown function retires the worker, and so does one in a destructor run while the registry is freed; three deadlines expiring in the shutdown window do not, while three handler fatals whose shutdown functions then outlive the deadline still do; and three thrown exceptions do not either. - Three of the seven are what actually discriminates, and they are the mixed ones. A block of three identical neutral outcomes passes just as well on a build that reads them as successes, because clearing the count does not retire a worker either — only a neutral outcome dropped into a run of fatals separates "did not count" from "wiped the count". - The destructor case reaches its fatal through a require of a helper that declares a class, because a class statement written inside a method is rejected while the file is compiled — which fatals at include time instead, in the handler, where the outcome is already counted and the test would have passed on a broken build. 29 integration tests.
diolektor
deleted the
fix/shutdown-function-fatal-resets-the-error-breaker
branch
August 13, 2026 21:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The consecutive-error breaker retires a worker after three failing requests in a row, and it decides what "failing" means from flags the request fiber's own
zend_catchraises. Everything that happens after the request handler returns is outside that arm, and PHP runs it behind guards of its own — so for the whole shutdown window the breaker was not merely blind, it was reading the wrong answer.A request that came apart there arrived at the check with no flag raised at all. That is the branch that reads a healthy worker and clears the run. So an application fataling in its shutdown function on every request was, to this mechanism, the picture of health, and one alternating between a fatal in its handler and a fatal in its shutdown function held the count at one for the life of the process. The same held for an exception thrown there.
What changed
Four outcomes at the end of a request are now classified the way the same outcome in the handler already was.
max_execution_timeexpiring in that windowThirteen lines of code in
ext/oxphp_fiber.c. The witnesses were already there and already being read: bothCG(unclean_shutdown)sites repair the abandoned frames by hand, which is the definition of a request that came apart, and the exception is already reported a few lines below.The deadline needs its own handling because
max_execution_timeis the one cancellation the interrupt handler never sees — the engine callszend_timeout()ahead of it. Every other cancellation is marked before it unwinds, so it survives whatever swallowed the bailout; this one is recognised by itsconnection_statusbit in the handler arm, which a shutdown-window bailout bypasses. Counting it would rotate the whole pool three requests at a time for an application that defers work to the end of a request and reaches a dependency that has gone slow.That bit is only read for a request that had not already come apart, and this is the subtle half. It is raised once and stands for the rest of the request, while the bailout flag beside it is lowered by every repair — so on its own it answers "did this request's deadline ever expire", not "did the deadline end this window". Without the guard, a handler fatal followed by a shutdown function outliving the remaining deadline would be filed as a cancellation, erasing a failure already established correctly, and handing any application a way to keep a permanently broken worker in the pool by registering one slow shutdown function.
Tests
Seven sequences appended to the worker-mode breaker profile, each ending in a probe that reads
Worker::current()->requestCount()together withoxphp_worker_recycles_by_reason_total{reason="error"}— 29 suite entries, taking the profile from 23 to 52.Three of the seven are what actually discriminates, and they are the mixed ones. A block of three identical neutral outcomes passes just as well on a build that reads them as successes, because clearing the count does not retire a worker either. Only a neutral outcome dropped into a run of fatals separates "did not count" from "wiped the count". The same asymmetry the profile already had between its cancellation blocks.
Every change in this PR was arrived at by a red test on the build immediately before it, in a chain of five images that differ by one edit each:
shutdown_retires,shutdown_does_not_clear,shutdown_dtor_retiresshutdown_timeout_neutralfatal_then_timeout_still_countsshutdown_throw_does_not_clearOne trap is worth repeating because the fixture looked right and was not. The destructor case first declared a class directly inside
__destruct; PHP rejects that while the file is compiled ("Class declarations may not be nested"), so the fatal happened at include time, inside the handler's own arm, where the outcome is already counted — the block passed on a broken build and only the log showed why. It now reachesE_COMPILE_ERRORat runtime through arequireof a helper that declares a class at top level. The point at which the destructor runs was confirmed separately with a non-fataling destructor writing markers: handler returning → shutdown function ran → destructor ran.Verification
breaker52/52default242/242,fibers32/32,timeout8/8,bailout2/2worker54/56 — the two errors areshared/test_channel_fanin_wakerandshared/test_channel_async_usevar_stress, known stress flakes; a control run of the same profile on an unmodified image produces the same twocargo fmt --check,cargo clippy --all-targets --no-default-features -D warnings,cargo test --no-default-features— all cleanNotes for review
exit()anddie()in a shutdown function are untouched: both throw an unwind exit rather than bailing out, so no flag is raised for them and such a request still counts as one that completed. The throw flag explicitly excludes unwind and graceful exit for the same reason.__toStringthat fatals while an exception is reported, and a userland output handler that fatals during the end-of-request flush. Same class, left out of this change deliberately and tracked separately.