http2: fix zombie session crash on socket close#61702
http2: fix zombie session crash on socket close#61702suuuuuuminnnnnn wants to merge 7 commits intonodejs:mainfrom
Conversation
|
Review requested:
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #61702 +/- ##
==========================================
- Coverage 89.75% 89.72% -0.03%
==========================================
Files 674 675 +1
Lines 204416 204530 +114
Branches 39285 39313 +28
==========================================
+ Hits 183472 183517 +45
- Misses 13227 13287 +60
- Partials 7717 7726 +9
🚀 New features to boost your workflow:
|
|
Hi @mcollina, I pushed a lint fix commit but CI didn’t re-run—could you please re-trigger the checks for this PR? |
Failed to start CI⚠ Commits were pushed since the last approving review: ⚠ - http2: fix zombie session crash on socket close ⚠ - http2: prevent assertion failure in OnStreamAfterWrite ⚠ - http2: handle write callback gracefully in zombie sessions ✘ Refusing to run CI on potentially unsafe PRhttps://github.com/nodejs/node/actions/runs/21752594207 |
pimterry
left a comment
There was a problem hiding this comment.
Thanks for the contribution @suuuuuuminnnnnn!
It looks like you've removed the fix that was described in the README here as part of your changes (in 6da4fe9) and now in the complete changeset it's just disabling an assertion. Is that intentional?
In the test, I think there's a little more work required to properly reproduce the issue described. I've just checked, and it passes on my machine with current versions of Node, without your src changes. The test should fail without your fix, and then pass once the fix is applied.
It would also be good to avoid the 1 second setTimeout here - timeouts like that are generally a fragile solution to race conditions and collectively they make our test suite much slower. It's better to use events & callbacks to cleanup at the correct time instead of guessing a duration. Once you have a failing test & working fix for it, let me know if you need help finding a good approach to avoid that.
|
Thanks! @pimterry
Let me know if you want the fallback timer removed entirely — I can tighten it further. |
pimterry
left a comment
There was a problem hiding this comment.
@suuuuuuminnnnnn The new test still passes for me locally, without your changes. We need a test that fails on current Node, and then a change that fixes it. I'm testing with Node v25.6.0 (latest release) and v24.13.0, but I suspect it works on all recent versions. If it does fail for you, can you confirm exactly what version you're using and how you're running the test?
This is also a bit unusual that there's multiple 'success' scenarios in the test code here: async error event, sync thrown error, or no error at all. We would usually want just one behaviour, unless there's a very good reason for unpredictable results. In my case I don't see the error happening at all in any cases, even if I run the test in a loop a few hundred times - it always just exits cleanly with no response.
I think it really should error in this case somehow, because the request really has failed (the socket is unexpectedly dead) and we should make that clear. From the description, it sounds like you're saying that we can reliably detect this failure case after a write completes, so we should be able to enforce that: make the request write some data, use the correct after-write checks to detect the issue, and raise an 'error' event if those checks fail.
If that's correct, that approach would mean we always emit an error async, in which case we can remove the other cases from the test: use mustCall on the 'error' handler, and remove the catch. We could then also run the cleanup code at that point (e.g. inside the 'error' handler) so we could remove the setTimeout completely.
Of course, I haven't dug into this as much as you, so I may be missing some details that make this impossible. If so can you explain more about what's happening here, and the process of what you've tried for detecting this issue, and what's working or not?
|
@pimterry Thanks for the review. I tried hard to turn this into a fail-first regression test, but I can’t reproduce any failure on current Node (v25.6.0 / v24.13.0), and the test keeps passing even without the src changes. That means I don’t have a valid regression test here, so I’m going to drop this PR for now. If someone has a known reproduction on current releases (exact script/steps), I’m happy to re-open with a deterministic failing test + fix. |
Summary
This PR fixes a crash caused by HTTP/2 zombie sessions when the underlying socket is dead but the HTTP/2 session remains “alive” in Node.js.
closeevent (e.g. packet drop “black hole”).CHECK(is_write_in_progress())inHttp2Session::OnStreamAfterWriteCHECK(!current_write_)inTLSWrap::DoWritenread < 0) inHttp2Session::OnStreamRead.Fixes: #61304
What is the current behavior?
When the network enters a “black hole” state (packets dropped without RST/FIN), the OS may consider the TCP socket dead/closed, but Node.js can fail to observe the close.
In this case:
session.closed === falsesession.destroyed === falsesession.state.outboundQueueSizeincreases continuously)Relevant code path (before):
What is the new behavior?
On read error (
nread < 0), we still notify the previous listener, then close the HTTP/2 session to prevent zombie state and subsequent assertion crashes.Key points:
Close()is idempotent / safe to call redundantly.Why is this correct?
A negative
nreadindicates an error/EOF condition at the stream/socket read layer. Keeping the HTTP/2 session alive after a read error allows the session to continue queueing outbound frames while the transport is effectively dead, which eventually leads to inconsistent internal write state and assertions.Closing the session on read error restores the expected lifecycle invariant: dead transport ⇒ closed session.
How was this tested?
New test (CI-friendly)
Adds a new parallel test that reproduces the “zombie” behavior without OS firewall rules by destroying the underlying socket directly:
client[kSocket].destroy()to kill the transport.New file:
test/parallel/test-http2-zombie-session-61304.jsCommands
Built locally (macOS arm64):
Ran test:
Spot-checked a related HTTP/2 test:
Additional context / reproduction (original issue)
The original issue can be reproduced reliably on macOS using pf firewall rules to drop packets (reported 100% reproduction), creating a network “black hole” where the socket becomes dead without Node observing a clean close.
Issue: #61304
Files changed
src/node_http2.cc(small change: close session onnread < 0)test/parallel/test-http2-zombie-session-61304.js(new)