Skip to content

Hand a channel's free room to a sender as state, not as a signal - #307

Merged
diolektor merged 1 commit into
mainfrom
fix/channel-async-stress-stuck-leaks-permits
Aug 16, 2026
Merged

Hand a channel's free room to a sender as state, not as a signal#307
diolektor merged 1 commit into
mainfrom
fix/channel-async-stress-stuck-leaks-permits

Conversation

@diolektor

Copy link
Copy Markdown
Contributor

A fiber blocked in Shared\Channel::send() could stay parked for good on a channel that was empty and open. send() arms no timer, so such a fiber never comes back: it holds its async-pool slot for the life of the process, and the pool runs that much smaller for everything after it. sendTimeout() burns its whole budget and reports Timeout against a channel that plainly has room.

What was wrong

A parked sender is woken by one thing only — a slot being freed — and that signal was addressed to whoever was already on the waiter list. Three ways to fall outside it:

  1. A freed slot that found the list empty. A sender joins the list a moment after its try_send fails, so a consumer that emptied the channel inside that moment woke nobody. If the room was still free by the time the sender finished parking it found the room for itself; if another producer had taken it by then — which is what several producers on one channel do constantly — the wake was gone rather than owed, because the next one belongs to the next freed slot. Every swallowed signal stranded one sender permanently. Measured on a stand emulating both fiber peers of one channel (4 producers, 1 consumer, capacity 1): 4025 parks, 4023 wakes, 2 signals delivered to an empty list, 2 senders parked forever.

  2. Room left behind rather than freed. try_send hands its value straight to a parked receiver, taking no slot, so the room a woken sender was given stays free and no further slot is ever freed to pass it on. Once the consumer outruns its producers and stays parked, every later send takes that direct route, the buffer never fills again, and everyone still parked stays parked. No race needed.

  3. Parking against a close. close() sweeps the waiter list exactly once; an id pushed after that sweep is one nobody resolves. This is the one shape that needs no second producer — a lone producer and whoever closes the channel are enough.

Shared\Channel used from ordinary threads rather than fibers was never affected: a thread blocked on a full channel polls it instead of waiting to be told.

What changed

All in src/plugins/ox_shared/types/channel.rs.

  • Free room is handed over as state, not as a one-off signal. send_waiters becomes a struct holding the parked ids and a credit counter under one lock. A free slot that finds nobody parked banks a credit; a sender redeems the credit instead of parking. Taking the credit and pushing the id are the same critical section as the freeing side, so whichever runs second sees what the first left.
  • A send that took no slot passes its untouched room on the same way a freed slot does.
  • The park-or-not decision, closure included, is taken under the waiter lock — the lock the close sweep takes — so a sender either lands before the sweep or is told the channel is closed.
  • The occupancy these decisions read comes from the buffer and the front-stash rather than from the pending gauge, whose increment lands after the deposit it counts and is therefore briefly wrong in both directions.
  • The credit counter is a debt of unclaimed frees, not a count of free slots: it is not spent when the room is taken again, and its ceiling is the channel capacity so it cannot grow without bound on a workload with no parked senders. The stale-credit cost — a resolved sender that finds the channel full again and parks anew — is tracked separately as a performance follow-up.

Verification

Every claim above is pinned by a test that was observed failing before it passed:

  • register_send_waiter_fires_when_the_freed_slot_was_taken_again, send_straight_to_a_receiver_passes_its_room_to_a_parked_sender and a_sender_parking_as_the_channel_closes_is_told_that_it_closed — deterministic, one per mechanism. Against an emulation of the unfixed protocol: 97 passed, 4 failed (these three plus the contention test).
  • Moving the closed check back outside the waiter lock, with everything else as merged, fails exactly one test: 100 passed, 1 failed.
  • room_left_behind_reaches_a_sender_that_was_still_on_its_way_to_park pins the banking rule rather than a past defect. It passes on the unfixed protocol, which re-tested the channel for room at park time and found that room, and fails against a variant that peeks at the waiter list and skips the bank. Its doc comment says so.
  • fiber_senders_are_never_stranded_on_an_empty_channel drives four producers and one consumer over a channel of capacity 1 through the park/wake protocol the fibers use, reporting the channel's state instead of hanging when a sender is stranded. It caught the second mechanism in 8 runs out of 10 before the fix.

Host: cargo fmt --check, cargo clippy --all-targets and cargo clippy --features plugin-shared,php --tests clean; 987 tests without default features, 1385 with plugin-shared, channel module 101/101 over three consecutive runs.

Worker test profile, images built with the legacy builder from the branch and from its base:

before after
profile, 3-4 runs 53-54 passed, 2-3 errors 55 passed, 1 error
channel_fanin_waker ERROR in 4 runs of 4 PASS in 3 of 3
the same test alone FAIL stuck got=1998 in 12.1 s OK in 0.64 s
cascade onto neighbours a rotating victim every run none

The one remaining error, shared/test_channel_async_usevar_stress, is not a lost wakeup and is not fixed here: its shape is one producer and one consumer, a parameterised copy passes identically on both builds (n=5000 in 4.2 s, n=20000 in 16.7 s), which puts one cross-thread handoff at ~0.84 ms — the async pool's idle backoff — so its n=50000 cannot fit the request deadline. That cost and the permit its killed peer leaves behind are tracked separately.

Fix:
  - A fiber blocked in `Shared\Channel::send()` could stay parked for good on a channel that was empty and open. It is woken only by a slot being freed, and that signal reached only whoever was already on the waiter list — a sender joins that list a moment after its `try_send` fails, so a consumer that emptied the channel inside that moment woke nobody. The sender still found that room for itself unless another producer had taken it first, and where one had, the wake was gone rather than owed: the next belongs to the next freed slot, and every swallowed signal stranded one sender permanently. Measured on a stand emulating both fiber peers of one channel: 4025 parks, 4023 wakes, 2 signals delivered to an empty list, 2 senders parked forever. A freed slot that finds nobody parked is now banked as a credit inside the same lock the waiter list lives in, and a sender redeems it instead of parking, so whichever side runs second sees what the first left.
  - The second way to lose it needed no race. `try_send` hands its value straight to a parked receiver, taking no slot — so the room a woken sender was given stays free, and no further slot is ever freed to pass it on. Once the consumer outruns its producers and stays parked, every later send takes that same direct route, the buffer never fills again, and everyone still parked stays parked. That path now hands its untouched room on the same way a freed slot does.
  - A sender that started parking while another fiber closed the channel also waited forever: `close()` sweeps the waiter list once, and an id pushed after the sweep is one nobody resolves — with no timer behind it, since `send()` arms none. The closed check now happens under the waiter lock, the lock that sweep takes, so the sender either lands before it or is told the channel is closed. This is the one shape that needs no second producer: a lone producer and whoever closes are enough.
  - The occupancy these decisions read comes from the buffer and the front-stash rather than from the `pending` gauge, whose increment lands after the deposit it counts and is therefore briefly wrong in both directions.
  - What an application saw: `sendTimeout()` burning its whole budget and reporting `Timeout` against a channel that had room, and `send()` never returning while holding its async-pool slot for the life of the process. In the worker test profile the fan-in channel test failed on every run before this and passes on every run after, and the cascade it caused onto neighbouring async tests is gone.

Tests:
  - Three deterministic tests, one per way a sender was stranded: a freed slot taken again before the sender parked, a send that reached a receiver directly, and a sender parking while the close sweep ran. All three fail against an emulation of the unfixed protocol, and the close one is the only test in the file that fails when the closed check alone is moved back outside the waiter lock.
  - A fourth pins the banking rule rather than a past defect — room that finds an empty waiter list is remembered instead of dropped. It passes on the unfixed protocol, which re-tested the channel for room at park time and found that room, and fails against the variant that peeks at the list and skips the bank.
  - A contention test drives four producers and one consumer over a channel of capacity 1 through the park/wake protocol the fibers use, reporting the channel's state instead of hanging when a sender is stranded. It caught the second defect in 8 runs out of 10 before the fix.

1385 tests (unit).
@diolektor
diolektor merged commit ca57b88 into main Aug 16, 2026
7 checks passed
@diolektor
diolektor deleted the fix/channel-async-stress-stuck-leaks-permits branch August 16, 2026 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant