Hand a channel's free room to a sender as state, not as a signal - #307
Merged
Merged
Conversation
Fix: - A fiber blocked in `Shared\Channel::send()` could stay parked for good on a channel that was empty and open. It is woken only by a slot being freed, and that signal reached only whoever was already on the waiter list — a sender joins that list a moment after its `try_send` fails, so a consumer that emptied the channel inside that moment woke nobody. The sender still found that room for itself unless another producer had taken it first, and where one had, the wake was gone rather than owed: the next belongs to the next freed slot, and every swallowed signal stranded one sender permanently. Measured on a stand emulating both fiber peers of one channel: 4025 parks, 4023 wakes, 2 signals delivered to an empty list, 2 senders parked forever. A freed slot that finds nobody parked is now banked as a credit inside the same lock the waiter list lives in, and a sender redeems it instead of parking, so whichever side runs second sees what the first left. - The second way to lose it needed no race. `try_send` hands its value straight to a parked receiver, taking no slot — so the room a woken sender was given stays free, and no further slot is ever freed to pass it on. Once the consumer outruns its producers and stays parked, every later send takes that same direct route, the buffer never fills again, and everyone still parked stays parked. That path now hands its untouched room on the same way a freed slot does. - A sender that started parking while another fiber closed the channel also waited forever: `close()` sweeps the waiter list once, and an id pushed after the sweep is one nobody resolves — with no timer behind it, since `send()` arms none. The closed check now happens under the waiter lock, the lock that sweep takes, so the sender either lands before it or is told the channel is closed. This is the one shape that needs no second producer: a lone producer and whoever closes are enough. - The occupancy these decisions read comes from the buffer and the front-stash rather than from the `pending` gauge, whose increment lands after the deposit it counts and is therefore briefly wrong in both directions. - What an application saw: `sendTimeout()` burning its whole budget and reporting `Timeout` against a channel that had room, and `send()` never returning while holding its async-pool slot for the life of the process. In the worker test profile the fan-in channel test failed on every run before this and passes on every run after, and the cascade it caused onto neighbouring async tests is gone. Tests: - Three deterministic tests, one per way a sender was stranded: a freed slot taken again before the sender parked, a send that reached a receiver directly, and a sender parking while the close sweep ran. All three fail against an emulation of the unfixed protocol, and the close one is the only test in the file that fails when the closed check alone is moved back outside the waiter lock. - A fourth pins the banking rule rather than a past defect — room that finds an empty waiter list is remembered instead of dropped. It passes on the unfixed protocol, which re-tested the channel for room at park time and found that room, and fails against the variant that peeks at the list and skips the bank. - A contention test drives four producers and one consumer over a channel of capacity 1 through the park/wake protocol the fibers use, reporting the channel's state instead of hanging when a sender is stranded. It caught the second defect in 8 runs out of 10 before the fix. 1385 tests (unit).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A fiber blocked in
Shared\Channel::send()could stay parked for good on a channel that was empty and open.send()arms no timer, so such a fiber never comes back: it holds its async-pool slot for the life of the process, and the pool runs that much smaller for everything after it.sendTimeout()burns its whole budget and reportsTimeoutagainst a channel that plainly has room.What was wrong
A parked sender is woken by one thing only — a slot being freed — and that signal was addressed to whoever was already on the waiter list. Three ways to fall outside it:
A freed slot that found the list empty. A sender joins the list a moment after its
try_sendfails, so a consumer that emptied the channel inside that moment woke nobody. If the room was still free by the time the sender finished parking it found the room for itself; if another producer had taken it by then — which is what several producers on one channel do constantly — the wake was gone rather than owed, because the next one belongs to the next freed slot. Every swallowed signal stranded one sender permanently. Measured on a stand emulating both fiber peers of one channel (4 producers, 1 consumer, capacity 1): 4025 parks, 4023 wakes, 2 signals delivered to an empty list, 2 senders parked forever.Room left behind rather than freed.
try_sendhands its value straight to a parked receiver, taking no slot, so the room a woken sender was given stays free and no further slot is ever freed to pass it on. Once the consumer outruns its producers and stays parked, every later send takes that direct route, the buffer never fills again, and everyone still parked stays parked. No race needed.Parking against a close.
close()sweeps the waiter list exactly once; an id pushed after that sweep is one nobody resolves. This is the one shape that needs no second producer — a lone producer and whoever closes the channel are enough.Shared\Channelused from ordinary threads rather than fibers was never affected: a thread blocked on a full channel polls it instead of waiting to be told.What changed
All in
src/plugins/ox_shared/types/channel.rs.send_waitersbecomes a struct holding the parked ids and a credit counter under one lock. A free slot that finds nobody parked banks a credit; a sender redeems the credit instead of parking. Taking the credit and pushing the id are the same critical section as the freeing side, so whichever runs second sees what the first left.pendinggauge, whose increment lands after the deposit it counts and is therefore briefly wrong in both directions.Verification
Every claim above is pinned by a test that was observed failing before it passed:
register_send_waiter_fires_when_the_freed_slot_was_taken_again,send_straight_to_a_receiver_passes_its_room_to_a_parked_senderanda_sender_parking_as_the_channel_closes_is_told_that_it_closed— deterministic, one per mechanism. Against an emulation of the unfixed protocol: 97 passed, 4 failed (these three plus the contention test).room_left_behind_reaches_a_sender_that_was_still_on_its_way_to_parkpins the banking rule rather than a past defect. It passes on the unfixed protocol, which re-tested the channel for room at park time and found that room, and fails against a variant that peeks at the waiter list and skips the bank. Its doc comment says so.fiber_senders_are_never_stranded_on_an_empty_channeldrives four producers and one consumer over a channel of capacity 1 through the park/wake protocol the fibers use, reporting the channel's state instead of hanging when a sender is stranded. It caught the second mechanism in 8 runs out of 10 before the fix.Host:
cargo fmt --check,cargo clippy --all-targetsandcargo clippy --features plugin-shared,php --testsclean; 987 tests without default features, 1385 withplugin-shared, channel module 101/101 over three consecutive runs.Worker test profile, images built with the legacy builder from the branch and from its base:
channel_fanin_wakerstuck got=1998in 12.1 sOKin 0.64 sThe one remaining error,
shared/test_channel_async_usevar_stress, is not a lost wakeup and is not fixed here: its shape is one producer and one consumer, a parameterised copy passes identically on both builds (n=5000in 4.2 s,n=20000in 16.7 s), which puts one cross-thread handoff at ~0.84 ms — the async pool's idle backoff — so itsn=50000cannot fit the request deadline. That cost and the permit its killed peer leaves behind are tracked separately.