Encode form and digest input without CharBuffer - #2385

Summary

• StringUtils.charSequence2Bytes / charSequence2ByteBuffer no longer run charset.encode(CharBuffer.wrap(cs)). They are on the path of every url-encoded form body (HttpUtils.urlEncodeFormParams) and every Digest hash (Realm), and of the Digest Authorization header (AuthenticatorUtils).
• The bytes are the same as before. The encoding step takes 25 to 50% less time on these paths and allocates the same or less, except for UTF-8 Digest input with non-ASCII credentials or realm.
• No public signature changes. Follows Encode setBody(String) bodies with String.getBytes #2371, which did the same for setBody(String).

Problem

CharBuffer.wrap(CharSequence) returns a buffer with no backing array, so the JDK encoder walks it char by char through virtual get/put calls and misses the array loop and the String intrinsics. All three callers pass a pooled StringBuilder, on the caller thread, once per request attempt (form) or three times per Digest response.

Change

charSequence2Bytes:

• a String goes through String.getBytes(charset);
• a StringBuilder (or any other CharSequence) in UTF-8, ISO-8859-1 or US-ASCII is copied char by char into an exact-size byte[] while every char is below a limit under which the charset maps a char to that byte: U+0100 for ISO-8859-1, U+0080 for UTF-8 and US-ASCII. On the first char at or above the limit it falls back to toString().getBytes(charset);
• any other charset goes through toString().getBytes(charset).

charSequence2ByteBuffer wraps that array. A url-encoded form is always ASCII (everything else is percent-encoded), so form bodies never reach the fallback. The Digest hash input (Realm) and the Digest Authorization header (AuthenticatorUtils.computeRealmHeader) reach it in UTF-8 (charset=UTF-8) when the credentials, realm or uri hold a char at or above U+0080, and in ISO-8859-1, the default Digest charset, only when they hold a char ISO-8859-1 cannot map (at or above U+0100), so Latin-1 credentials such as accented letters stay on the loop.

StringBodyEncodeBenchmark.charsetEncode from #2371 measured the old path through charSequence2ByteBuffer(String, UTF_8), which now goes to getBytes too; it now calls UTF_8.encode(CharBuffer.wrap(body)) directly, so it keeps measuring the old path.

String.getBytes(charset) and Charset.encode both replace malformed and unmappable input with the encoder's replacement and write the same byte order marks; #2371 checked this for every charset. The new tests check it again here:

• StringUtilsTest: both methods against charset.encode(CharBuffer.wrap(cs)) for every charset the JDK can encode, 17 edge inputs (unpaired surrogates, supplementary chars, BOM, replacement char, NUL/DEL/U+0080/U+00FF/U+0100, and U+0080 alone and between ASCII chars, so each limit is checked on its own: with > 0x80, or with the ISO-8859-1 limit at 0x101, the test fails), and four kinds of CharSequence: String, StringBuilder, a StringBuilder with a UTF-16 coder (the state a pooled builder keeps after it once held a char above U+00FF) and CharBuffer.
• HttpUtilsTest: urlEncodeFormParams against US_ASCII.encode of the same url-encoded form, in UTF-8, ISO-8859-1 and GBK, with non-ASCII, a supplementary char, unpaired surrogates and a null value, from a pooled builder with a UTF-16 coder.
• RealmTest: the Digest response for Cyrillic user, a password with a lone surrogate, Euro sign and emoji, and a non-ASCII realm, in ISO-8859-1 (default) and UTF-8 (charset=UTF-8), against MD5 over charset.encode bytes.

The returned ByteBuffer now has capacity equal to its limit. Before, it was whatever the encoder had allocated (1.1 bytes per char for UTF-8, more after growth). That capacity is not specified by Charset.encode, and every caller reads only position to limit.

An alternative, sb.toString().getBytes(charset) for everything, is faster still but copies the builder into a String first, which roughly doubles allocation per form body against main (numbers below), so it was not taken.

Benchmark

FormDigestEncodeBenchmark (new) calls production code only, so the same class runs against main and this branch. JDK 17, -f 2 -wi 4 -i 5 -prof gc, three rounds alternating main and branch; the median round is shown.

The end-to-end form and Digest timings are dominated by url-encoding and by SecureRandom for the cnonce. On this shared host they moved by more than the change itself between runs of the same build (main digestAscii measured anywhere from 1.8 to 9.2 us), so only their B/op is reported. B/op is deterministic up to a few dozen bytes.

encodeUtf8DigestBuilder allocates more: the loop's array is discarded at the first non-ASCII char, and getBytes(UTF_8) on a UTF-16 string uses a 3-bytes-per-char scratch array, the same trade-off as in #2371. In ISO-8859-1, the default Digest charset, the loop covers every char up to U+00FF, so the fallback, with its discarded array and String copy, is left to credentials, realm or uri holding a char ISO-8859-1 cannot map (it becomes ?, as before). Either way this is short Digest input only; ASCII Digest input and every form body stay on the loop. The ISO-8859-1 limit was raised from U+0080 to U+0100 after review; the numbers above were measured before that and do not touch it (encodeDigestBuilder is ASCII).

Loop against toString().getBytes (exploratory, not committed), the encoding step alone, UTF-8, -f 2 -wi 4 -i 5 -prof gc, ns/op and B/op:

The host was shared with other builds, so take the timings as indicative.

Second-agent review

Both reviewers ran read-only, in fresh sessions, on git diff upstream/main...HEAD.

• Guided reviewer, checklist AGENTS.md: Codex gpt-6-astra, reasoning effort medium.
• Independent reviewer, no checklist: Claude Code claude-opus-5-5, effort medium, tools Read/Grep/Glob only.

Maintainer review on 665746074, addressed in a763e6bc9 and 253d17b3c:

• c > 0x80 passed every test: U+0080 alone and a U+0080 b were added to StringUtilsTest; the mutant now fails.
• StringBodyEncodeBenchmark.charsetEncode measured getBytes through charSequence2ByteBuffer: it now calls UTF_8.encode(CharBuffer.wrap(body)).
• The fallback also runs for ISO-8859-1, the default Digest charset: it now runs there only for chars ISO-8859-1 cannot map (round 4 (a)), and the description says so.

AI disclosure

Claude Code on behalf of Pavel Ptashyts. The commits include Co-Authored-By per AGENTS.md.

Test plan

• StringUtilsTest, HttpUtilsTest, RealmTest, AuthenticatorUtilsTest, DigestAuthTest, DigestAuthRfc7616Test, ParamEncodingTest: 94 tests pass.
• mvnw.cmd -B -ntp -Dgpg.skip=true clean verify on JDK 17 (no JDK 11 on the host): 1,998 tests, 0 failures, 0 errors, 35 skipped, on 665746074 and again on 253d17b3c; Revapi passed. 9dd49bd8b changes only a benchmark Javadoc (JMH sources are not built by Maven). Source/target is 11. JDK 11 coverage comes from CI.
• Mutation checks: c > 0x80 and an ISO-8859-1 limit of 0x101 each make StringUtilsTest fail.
• StringBodyEncodeBenchmark and FormDigestEncodeBenchmark compile with javac --release 11.

Claude Code on behalf of Pavel Ptashyts

🤖 Generated with Claude Code