Skip to content

fix(netty): never let a queued suspend override a later resume - #2361

Merged
hyperxpro merged 3 commits into
AsyncHttpClient:mainfrom
mkurz:fix/suspend-resume-order
Nov 5, 2026
Merged

hyperxpro merged 3 commits into
AsyncHttpClient:mainfrom
mkurz:fix/suspend-resume-order

Conversation

@mkurz

@mkurz mkurz commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • A ResponseBodyControl.suspend() queued from another thread no longer takes effect after a resume() that the event loop made in the meantime, which could leave a response suspended with nothing left to resume it.
  • A resume() is still always applied, so it is never lost to a suspend() that a callback makes after it was queued.
  • Document when control calls take effect. No API change.

Problem

NettyResponseBodyControl applies a call made on the event loop before it returns, and queues a call made on any other thread to the event loop. Queued calls take effect in the order they were queued, without regard to calls the event loop made inline in the meantime.

The stall:

  1. A consumer thread calls suspend(), which is queued.
  2. Before it runs, a body callback on the event loop calls resume(), which takes effect immediately.
  3. The queued suspend() then runs and suspends reads again.

Nothing calls resume() after that, so the response stalls until the request timeout. The callback's resume() was the latest decision, but the older suspend() overrode it.

This pattern occurs in reactive consumers that suspend from whichever thread finds the buffer full. Play WS hit a variant of it, and avoids it today by suspending only on the event loop.

Change

  • suspend() records a suspend request, and resume() clears it. On the event loop, suspend() suspends at once. From another thread, it queues the suspension.
  • A queued suspension is skipped if, when it would take effect, the most recent call to suspend() or resume() was a resume.
  • resume() is always applied, and calls made on the event loop take effect before they return, as before.

This is not "the latest call wins". Only suspensions can be skipped. Skipping a resume instead would reintroduce a lost-wakeup stall: a consumer takes the last buffered part and calls resume(), a callback on an older view calls suspend() before that resume is applied, and the resume must still win. A test pins this case.

Calls made concurrently on different threads are not ordered beyond these rules. For example, with suspend, resume and suspend queued from other threads, the first suspension still takes effect briefly, because the most recent call is a suspend when it runs; the queued resume and the second suspension then follow.

The ResponseBodyControl Javadoc now states when calls take effect:

  • a call on the event loop takes effect before it returns;
  • calls from other threads take effect later on the loop, possibly after further callbacks;
  • a resume() always takes effect;
  • a suspend() from another thread is skipped if the most recent call was a resume;
  • concurrent calls on different threads are not ordered otherwise.

Compatibility

No public API change. Behavior changes only for a suspend() from another thread whose queued suspension is overtaken by a resume(): it no longer takes effect. All other call orders behave as before.

Related

The separate pull request #2359, "feat: add ResponseBodyControl.execute to decide in sequence with body delivery", lets a consumer make its suspend/resume decision on the event loop in the first place. The two changes are independent and merge cleanly with each other.

AI disclosure

Claude Code on behalf of Matthias Kurz. The commits include Co-Authored-By per AGENTS.md.

Test plan

New tests, for HTTP/1.1, for HTTP/2, and as unit tests in NettyResponseBodyControlTest (with Netty auto-read on and off):

  • an earlier suspend() from another thread does not undo a later resume();
  • a resume() from another thread is not lost to a later suspend().

NettyResponseBodyControlTest also covers:

  • sequences of calls queued from another thread: suspend, resume, suspend ends suspended, and resume, suspend, resume ends resumed, with the exact number of suspensions started and ended;
  • a suspend() after a skipped one still takes effect;
  • a suspend() on the event loop takes effect even if a resume() from another thread clears its request between recording and acting on it. The channel's event loop is a proxy whose inEventLoop() runs that resume at exactly that moment.

The unit tests also check that the suspension start/end hooks stay balanced and that exactly one read is requested per resume. Calls from another thread go through a single-thread executor owned by each test, not the common pool.

Broken variants, each run against the 31 control tests:

  • A queued suspension always applied, as on main: 7 failures and 1 error (HTTP/2, by timeout). These are the tests that an earlier suspend() must not undo a later resume(), the sequence test, and the test of a suspension after a skipped one.
  • The latest call always wins, so a queued resume() is dropped for a later suspend(): 5 failures and 1 error (HTTP/2, by timeout). These are the tests that a resume() must not be lost, and the sequence test.
  • A skipped suspension that still runs the suspension start hook: 4 failures, all in NettyResponseBodyControlTest, through the counts of started suspensions. The integration tests don't catch this variant.
  • A suspend() on the event loop that still goes through the queued-suspend check, as in the first revision: 1 failure, the race test.

Runs:

  • ResponseBodyControlTest, Http2ResponseBodyControlTest and NettyResponseBodyControlTest on JDK 11: 31 tests passed.
  • ./mvnw -B -ntp -Dgpg.skip=true clean verify on JDK 11: 1,804 tests, 0 failures, 0 errors, 11 skipped; Revapi passed. No test-skipping flags were used.
  • The same 31 tests, compiled on JDK 11 and run on JDK 17, 21 and 25 (Maven Surefire's -Djvm): 0 failures on each.
  • All four open AHC changes merged together (raw Cookie headers on retries, ResponseBodyControl.execute, suspend/resume order, demand-bounded decompression): they merge without conflicts, together and in every pair, and ./mvnw -B -ntp -Dgpg.skip=true clean install on JDK 11 passes with 1,861 tests, 0 failures, 0 errors, 11 skipped; Revapi passed. Rerun with the review fixes of this pull request and fix(netty): stop inflating a suspended HTTP/1.1 response #2362.

A ResponseBodyControl call made off the event loop is queued and takes
effect later on the loop, in the order the calls were queued. A
suspend() queued from another thread could therefore take effect after
a resume() that the event loop made in the meantime, for example from a
body callback, and leave reads suspended with nothing left to resume
them.

Record a suspend request when suspend() is called and clear it when
resume() is called. A queued suspend() is skipped if, when it would
take effect, the most recent call was a resume(). A resume() is always
applied, so it is never lost to a suspend() that a callback made after
it was queued. Calls made on the event loop take effect before they
return, as before. Calls made concurrently on different threads are
not ordered beyond that.

Document when control calls take effect.

Claude Code on behalf of Matthias Kurz

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mkurz and others added 2 commits October 22, 2026 01:27
Motivation:
suspend() recorded the suspend request and then ran applySuspend,
inline when called on the event loop. A resume() from another thread
could clear the request in between, and the suspend() on the event
loop then did nothing, although a call made on the event loop is
documented to take effect before it returns.

Modification:
On the event loop, suspend() now suspends directly. Only a suspend()
from another thread is queued, and only a queued one can be skipped.
A resume() queued from another thread still takes effect afterwards.
Keep a single comment on the skip rule, next to the check.

Result:
A suspend() on the event loop always takes effect before it returns.
A test runs a resume() from another thread at the moment suspend()
asks whether it is on the event loop; it fails without this change.

Claude Code on behalf of Matthias Kurz

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Assert the exact number of suspensions started and ended for the
  sequences of queued calls, so that test fails both on main and if
  the latest call always won.
- Explain why the tests that a resume is not lost to a later suspend
  pass without skipping too: they fail if a resume is skipped.
- Call from another thread through a single-thread executor owned by
  the test instead of the common pool, which can be busy on a small
  machine while the event loop waits for it.
- After the HTTP/2 test of an earlier suspend, check that a sibling
  stream on the same connection still gets its whole body, as the twin
  test does.

Claude Code on behalf of Matthias Kurz

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mkurz

mkurz commented Oct 22, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I've pushed two commits on top: the fix for a suspend() on the event loop, and the test changes. I also updated the description. Replies are inline.

@mkurz
mkurz requested a review from hyperxpro October 22, 2026 10:48
@hyperxpro
hyperxpro merged commit 0a9b14a into AsyncHttpClient:main Nov 5, 2026
13 checks passed
@mkurz
mkurz deleted the fix/suspend-resume-order branch November 5, 2026 20:26
@mkurz mkurz mentioned this pull request Nov 5, 2026
5 tasks done
pavel-ptashyts added a commit to maygemdev/async-http-client that referenced this pull request Nov 6, 2026
Brings in AsyncHttpClient#2359 and AsyncHttpClient#2361, which also change NettyResponseBodyControl.
The only conflict was the field block: keep the volatile int active
flag from this branch alongside the new suspendRequested flag.

Claude Code on behalf of Pavel Ptashyts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hyperxpro pushed a commit that referenced this pull request Nov 6, 2026
## Summary

- Upgrades Netty from 4.2.19.Final to 4.2.20.Final. Its [security
fixes](https://github.com/netty/netty/releases/tag/netty-4.2.20.Final)
include several in modules AHC uses: three out-of-bounds reads in
`netty-buffer`, unbounded memory usage in `netty-codec-http2`,
insufficient control character neutralization in `netty-codec-http`,
insufficient entropy in `netty-resolver-dns`, and an mTLS bypass via
session resumption in `netty-handler`.
- On 4.2.20, two of AHC's tests fail, both because of Netty changes.
This PR adapts AHC to them.

## Change

- **Upgrade Netty to 4.2.20.Final, and check the second gzip magic
byte:** `JdkZlibDecoder` now rejects a gzip header with a wrong second
magic byte (netty/netty#17972). The gzip inflate loop from #2357 follows
`JdkZlibDecoder`, and `Http1ContentDecompressorTest` compares the two,
so it now checks that byte too. Both bytes are read unsigned, since
`0x8b` is negative as a signed byte.
- **Keep zstd working at a non-positive threshold:** since
netty/netty#17896, the `ZstdDecoder` that `HttpContentDecompressor`
creates sizes its output buffers by
`io.netty.compression.defaultMaxForwardBytes`. At 0 it hands on an empty
body without an error; below 0 it fails with `IllegalArgumentException:
initialCapacity : -1`. AHC leaves zstd to that decoder while the
property is not positive (#2362), so it now replaces it with a
`ZstdDecoder` whose buffers hold `MAX_PART_SIZE`. The size limit still
applies, and the body is still decompressed eagerly there, as before. At
the default value of the property nothing changes. netty/netty#18289
fixes the decoder in Netty; this workaround is harmless with that fix
and can go once AHC requires a Netty version that has it.

## Notes

- Netty 4.2.0 to 4.2.4 have no `ZstdDecoder(int)` constructor.
Overriding AHC's Netty with one of those and also setting the property
to 0 or below would fail zstd responses with a `NoSuchMethodError`.

## AI disclosure

Claude Code on behalf of Matthias Kurz. Both commits include
`Co-Authored-By` per `AGENTS.md`.

## Test plan

- [x] `main` at `5542c9fc4` with `-Dnetty.version=4.2.20.Final`: 1,857
tests, 2 failures:
- `Http1ContentDecompressorTest.decodesLikeNettyForEveryBodyAndSplit`,
case "gzip second magic byte ignored": Netty now fails that body, AHC
did not.
- `Http1ContentDecompressorForwardThresholdTest`: "the zstd body
differs".
- [x] `Http1ContentDecompressorForwardThresholdTest` now runs at
thresholds 0 and -1, each in a fresh JVM, and also checks the size limit
for zstd and that every exchange releases its input and buffers. On
4.2.20 without the zstd commit it fails at both ("the zstd body
differs", "the zstd body did not end"). With it, it passes on 4.2.19 and
4.2.20.
- [x] `./mvnw -B -ntp -Dgpg.skip=true clean verify` on JDK 11, on `main`
`3e95ec8c2`: 1,905 tests, 0 failures, 0 errors, 11 skipped; Revapi
passed. No test-skipping flags were used.
- [x] `Http1ContentDecompressorForwardThresholdTest` and
`Http1ContentDecompressorTest` compiled on JDK 11 and run on JDK 17, 21
and 25 (Maven Surefire's `-Djvm`): 36 tests each, 0 failures.
- [x] This branch merged with #2369: `clean verify` on JDK 11, 1,914
tests, 0 failures, 0 errors, 11 skipped; Revapi passed (run before
#2359, #2360 and #2361 were merged, with those three merged in as well).

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants