Skip to content

Do not assume GetSpan honoured the size hint - #3220

Merged
mgravell merged 7 commits into
mainfrom
marc/getspan-hint-verify
Sep 13, 2026
Merged

mgravell merged 7 commits into
mainfrom
marc/getspan-hint-verify

Conversation

@mgravell

@mgravell mgravell commented Sep 13, 2026

Copy link
Copy Markdown
Collaborator

Fixes the intermittent ArgumentOutOfRangeException from MessageWriter.WriteUnifiedSpan reported in
this gist — and, while there, a
silent buffer overrun on a neighbouring path.

The bug

IBufferWriter<T>.GetSpan(sizeHint) guarantees one element, not sizeHint. The "should be at least
this size" wording in the docs is left over from when the parameter was called minSize; it is advisory,
and consuming code is expected to test what it got and fall back — which is exactly what the BCL's own
BuffersExtensions does, looping and copying in slices rather than demanding one contiguous block. See
CommunityToolkit/dotnet#1208.

MessageWriter treated it as a guarantee throughout: take the span, write the full amount into it,
advance by that amount.

The looser contract exists for transports with page limits, which can honour reasonable requests and
refuse excessive ones. CycleBuffer is one — it caps the hint at 1k, sizes a fresh segment from the
committed total rather than from the hint, and, the case with no floor at all, hands back a dangling
recycled segment exactly as it is, whatever length that happens to be:

var spare = endSegment.Next;
if (spare is not null) { ... return spare; }   // whatever size it happens to be

All legitimate. The callers were wrong.

That intermittency is specific to CycleBuffer — the recycled-spare path is what makes the reported
failure show up under concurrency rather than every time. It is worth separating out, because the other
under-delivering writer is not intermittent at all. BlockBuffer.GetBuffer clamps the hint to [16, 128]
and then returns whatever is left in the block, and says so itself:

// note this isn't an actual 'max', just a max of what we guarantee;
// we give the caller whatever is left in the buffer

So on BlockBuffer this is a straightforward capacity bug: any request above 128 is under-served whenever
the block has between 128 and 527 bytes left, which covers the whole quick-span path. BlockBufferUnderDeliversWithoutAnyContrivedWriter exercises it through TestHarness.Write with no synthetic writer
involved at all.

On main, that test does not throw — the test host dies with exit code 139 (SIGSEGV), reproduced
twice. That is the "silent overrun" characterisation, executed rather than reasoned.

The reported exception is the benign case

Every fixed-size write had the same shape, and one of them is worse than an exception:

var span = writer.GetSpan(expectedLength);
fixed (byte* bPtr = &MemoryMarshal.GetReference(span))
    Encoding.UTF8.GetBytes(cPtr, value.Length, bPtr, expectedLength);
writer.Advance(expectedLength);

expectedLength is passed to the encoder as the destination capacity rather than being derived from the
span, so against a short span that writes past the end of memory it does not own — silently. It now
falls through to the encoder loop immediately below it, which sizes every write from span.Length and
already copes with whatever it is given.

Shape of the fix

The hot sites check the length inline and hand the rare case to a separate NoInlining method that
composes in a stack buffer and writes through BuffersExtensions.Write.

That split is load-bearing rather than stylistic. A stackalloc in the caller's cold branch changes
codegen for the whole method: writing the fallbacks inline cost 20% on the simplest command shape. It
is the same "inline-optimized for it-fits, pathological case doesn't inline" structure the BCL uses, and
the issue above describes. The genuinely cold sites (WriteUnifiedUInt64, WriteUnifiedDouble,
WriteInteger, the SHA1 path) keep a PrefixScratch helper, which slices both spans to the requested
length so it bounds-checks in Release rather than relying on a Debug.Assert.

Sites: both WriteHeader overloads, both WriteMultiBulkHeader overloads, WriteUnifiedPrefixedString,
WriteUnifiedPrefixedBlob, WriteUnifiedInt64, WriteUnifiedUInt64, WriteUnifiedDouble,
WriteInteger, WriteUnifiedSequenceIterator, the SHA1 bulk-string path, WriteUnifiedSpan and
WriteCrlf. The encoder loop was already correct; the commented-out WriteUnifiedSequence is untouched.

Also fixed while in there

  • Length prefixes are sized for an int64. WriteMultiBulkHeader(long), WriteUnifiedPrefixedBlob
    (prefix.LongLength + value.LongLength) and WriteUnifiedSequenceIterator all format a long but asked
    for 3 + MaxInt32TextLen = 14, against a possible 23. The commented-out WriteUnifiedSequence had it
    right. Now covered by a test — but defensively, not as a live bug: the test fails on main only
    because it calls WriteMultiBulkHeader(long.MaxValue) directly, and no caller can reach that, since
    argument counts come from array lengths. The sizing follows the parameter type rather than today's
    callers; the test comment says the same.
  • Large binary values keep their length prefix in place. WriteUnifiedSpan sends everything over
    MaxQuickSpanSize (512) down the piecewise path because no single span could hold it — so for a 4KB
    byte[] that path is the normal one, not a fallback, yet it called WriteCountPrefixSlow
    unconditionally. That composed $4096\r\n into a stack buffer and handed it to a hintless
    BuffersExtensions.Write: an extra copy, and the same length-prefix-split-across-segments
    fragmentation this PR fixes two hunks earlier in WriteCrlf. The string path never had it
    (WriteUnifiedPrefixedString calls the checked WriteCountPrefix), so it was asymmetric for no reason.
    It now calls WriteCountPrefix too, and is renamed WriteUnifiedSpanPiecewise…Slow described only
    one of its two entry reasons. The two header fallbacks got the same treatment: the ask that was declined
    was larger than 23, so the smaller one may well still land in place. WriteCountPrefixSlow is now
    reached only from WriteCountPrefix.
  • WriteUnifiedInt64 passed the wrong scratch constant - MaxPrefixScratch (30) while requesting 27.
    It fit by three bytes, guarded only by a Debug.Assert that is gone in Release.
  • WriteCrlf keeps its hint of 2. Falling through to BuffersExtensions.Write would ask with no hint
    at all, and CycleBuffer's hint <= 0 branch returns anything non-empty - so a 1-byte segment tail
    would split a CRLF across segments rather than rolling to a fresh one. Protocol-correct either way, but
    a fragmentation change on a per-bulk-string path is not one to make by accident.

Cost

Formatting only, no server, via the added MessageWriterBenchmarks:

before after
SET key value 64.48 ns 66.42 ns +3.0%

That is the honest price of checking, and it is reproducible — measured twice independently, at +3.1% and
+3.0%.

Only that figure is claimed. The mixed-value and 4KB cases swung by several percent between runs in
both directions (main's own 4KB figure moved between 187ns and 204ns across sessions), so they are not
reported: BenchmarkDotNet's error bars measure within-run precision, not between-run reproducibility. The
filter set also matters — including the fallback benchmark in the same run moved SET key value to
70.6ns, so before-and-after runs must measure the same set.

The benchmark has two writers, because they answer different questions. Reusable always returns the
remainder of a 16KB buffer, so the hint is always honoured and the table above measures what the check
costs. Stingy never returns more than 8 bytes, so every burst takes the fallback; that one cannot be
compared against a pre-fix build, because a pre-fix build faults.

8 is deliberate, and an earlier 64 was useless: the largest ask in SET user:1 marc is 23 (WriteHeader
wants 9 framed command bytes + 3 + MaxInt32TextLen; WriteCountPrefix wants 3 + MaxInt64TextLen), so a
64-byte cap declines nothing and measures the direct path twice. 16 declines the two 23-byte asks but still
does not fault on main, whose WriteHeader writes only 13 of the 23 it asked for. 8 exercises every
fallback and faults on main — confirmed by running ShortSpansStillProduceCorrectFrames(max: 8), the
same command shape, against main.

Tests

BufferWriterHintTests uses a writer that honours at most N bytes per span and places a canary past the
span it hands out
, verified on Advance, on the next hand-out, and on reading the result — all three,
because a span written past and then abandoned without advancing would slip through a check on Advance
alone.

That detail matters: my first version of this harness used a plain growing buffer and passed against the
unfixed code
— an overrun lands harmlessly in the slack that follows, while the real writer corrupts the
next thing in the pool.

Direct coverage for the paths no message shape reaches: WriteSha1AsHex (the tightest fit in the file -
47 requested, exactly 47 written, no slack), keyspace-isolation prefixes, the prefixed multi-bulk header,
WriteInteger, and a long-width count.

34 tests. Against main, 16 of the 31 synthetic-writer tests fail — and the 3 BlockBuffer tests cannot
be counted alongside them, because they take the test host down with SIGSEGV before the run finishes.

Verification

Full suite against the docker topology: 6184 passed, 150 skipped, 1 failed. The failure is
FailoverTests.SubscriptionsSurvivePrimarySwitchAsync, which fails identically on main.
CommandTimeoutTests.DefaultHeartbeatLowTimeout and DatabaseTests.CountKeys appeared in one run, were
absent from the next, and pass consistently in isolation — flaky, not related. Full
dotnet build Build.csproj -c Release /p:CI=true passes.

Not addressed

CycleBuffer is unchanged, deliberately — it is within its rights, and the callers were the problem.

Separately noticed while investigating: ScriptReadOnlyCommandTests, ScriptEvalRespNoScriptTests and
RenderedArgsLeakTests fail rather than skip when no server is present, because they hand-roll their
connection to get a command-map hook (TestBase.Create has none) and so lose the
AbortOnConnectFail = false plus Assert.SkipUnless(conn.IsConnected, ...) that TestBase applies. Not
touched here, but it reports "broken" instead of "skipped" on any machine without Redis, which is a
misleading signal when you are trying to tell whether your own change broke something.

IBufferWriter<T>.GetSpan(sizeHint) guarantees one element, not sizeHint. The
"should be at least this size" wording in the docs is left over from when the
parameter was called minSize; it is advisory, and consuming code is expected to
test what it got and fall back. The BCL does exactly that - BuffersExtensions
loops and copies in slices rather than demanding one contiguous block. See
CommunityToolkit/dotnet#1208.

MessageWriter treated it as a guarantee throughout: take the span, write the
full amount into it, advance by that amount.

The looser contract exists for transports with page limits, which can honour
reasonable requests and refuse excessive ones. CycleBuffer is one - it caps the
hint at 1k, sizes a fresh segment from the committed total rather than from the
hint, and, the case with no floor at all, hands back a dangling recycled segment
exactly as it is, whatever length that happens to be. All legitimate; the
callers were wrong. It is also why this reproduces intermittently and under
concurrency rather than deterministically.

Reported as an ArgumentOutOfRangeException from WriteUnifiedSpan, but that is
the benign manifestation and not the only site. Every fixed-size write had the
same shape, and one of them is worse than an exception:

    var span = writer.GetSpan(expectedLength);
    fixed (byte* bPtr = &MemoryMarshal.GetReference(span))
        Encoding.UTF8.GetBytes(cPtr, value.Length, bPtr, expectedLength);
    writer.Advance(expectedLength);

expectedLength is passed to the encoder as the destination capacity rather than
being derived from the span, so against a short span that is a buffer OVERRUN -
it writes past the end of memory it does not own, silently. It now falls through
to the encoder loop below it, which sizes every write from span.Length and
already copes with whatever it is given.

Every other site checks the length inline and hands the rare case to a separate
NoInlining method that composes in a stack buffer and writes through
BuffersExtensions.Write. That split is load-bearing, not style: a stackalloc in
the caller's cold branch changes codegen for the whole method, and writing the
fallbacks inline cost 20% on the simplest command shape. It is the same
"inline-optimized for it-fits, pathological case doesn't inline" structure the
BCL uses.

Measured, formatting only, SET key value: 64.66ns before against 66.64ns after,
so +3.1% - small error bars, so real rather than noise. That is the honest price
of checking. Larger values come out ahead (203.7ns to 184.9ns), because sharing
one length-prefix helper removed duplicated work.

Sites fixed: both WriteHeader overloads, both WriteMultiBulkHeader overloads,
WriteUnifiedPrefixedString, WriteUnifiedPrefixedBlob, WriteUnifiedInt64,
WriteUnifiedUInt64, WriteUnifiedDouble, WriteInteger,
WriteUnifiedSequenceIterator, the SHA1 bulk-string path, WriteUnifiedSpan and
WriteCrlf. The encoder loop was already correct; the commented-out
WriteUnifiedSequence is left alone.

Tests use a writer that honours at most N bytes per span and places a canary
past the span it hands out, so an overrun is detected rather than landing
harmlessly in slack - a plain growing buffer passes while the real writer
corrupts the next thing in the pool. 13 tests across the value shapes; 6 fail
without the fix, with two distinct crash modes.

Full suite against the docker topology: 6172 passed, 150 skipped, 1 failed -
FailoverTests.SubscriptionsSurvivePrimarySwitchAsync, which fails identically on
main.
Restores the NoInlining split that was lost to a git checkout during
benchmarking, plus the review findings:

- WriteUnifiedInt64 passed MaxPrefixScratch (30) while requesting 27; now
  MaxValueScratch, which is what its doc comment says it is for.
- PrefixScratch slices both spans to length, so it bounds-checks in Release
  rather than relying on a Debug.Assert that is gone there.
- WriteCountPrefix is sized for an int64: the multi-bulk header, the blob prefix
  and the sequence iterator all format a long and previously asked for int32
  width, so the direct path could be handed exactly 14 bytes and write 23.
- WriteCrlf keeps its hint of 2 rather than falling to BuffersExtensions.Write,
  which asks with no hint at all and would split a CRLF across segments.
- The test harness checks its canary on hand-out and on reading the result, not
  only on Advance, so a span written past and then abandoned cannot slip through.
- Coverage for WriteSha1AsHex (the tightest fit in the file), keyspace prefixes,
  the prefixed multi-bulk header, WriteInteger, and a long-width count.
Adds a stingy writer to the benchmark, so the fallback branch is actually
exercised: the Reusable writer always returns the remainder of a 16KB buffer and
so only ever measures what the CHECK costs, which the previous table could not
be read as the cost of the fix generally.

Corrects the numbers, which were overstated in two directions:

- The -9.2% on LargeValue was partly measurement drift. main's own figure moved
  between 187ns and 204ns across sessions, so the claim that sharing a
  length-prefix helper made large values faster is not supported and is dropped.
- MixedValueShapes swung between -0.4% and +4.5% across runs. BenchmarkDotNet's
  error bars measure within-run precision, not between-run reproducibility, and
  at this scale the latter is several percent.

Only the SET key value figure is reproducible: 64.48ns against 66.42ns, so about
+3%, measured twice independently with matching filter sets. Note the filter set
matters - including the fallback benchmark in the same run moved the figure to
70.60ns, so before-and-after runs must measure the same set.
- add ShortSpansStillProduceCorrectFramesForLargeBinaryValues: byte[] takes a
  different route from string through WriteUnifiedSpan, and the value-shapes
  test only reached it with four bytes. 400/512/513 at caps of 8/400/520.
- note that [InlineData(520, 512)] on the string theory does not stress that
  path - a 512-byte string asks for exactly 512, which a 520-cap writer
  honours, so it passes unfixed. Kept as regression cover, not as evidence.
- relabel CountPrefixIsSizedForALong as defensive: main's long overload really
  does hint int32 width, but argument counts come from array lengths, so no
  caller can reach it.
- document that TryGetSpan returning false is not side-effect free, and the two
  ways StingyWriter is stricter than a real writer.
519/512 is the case that bites: the hint carries int32-text slack it rarely
uses, so a cap has to fall below the real frame length (1 + digits + 2 + len +
2 = 520 here), not merely below the hint, to overrun. Marked which cases fail
unfixed and which are boundary cover.
…ter stingy

Three findings from review:

1. WriteUnifiedSpanSlow was the NORMAL path for values over MaxQuickSpanSize,
   not a fallback, and called WriteCountPrefixSlow unconditionally - so every
   4KB byte[] composed its $4096\r\n into a stack buffer and handed it to a
   hintless Write, costing a copy and risking the prefix splitting across
   segments: the same fragmentation just fixed in WriteCrlf. The string path
   never had this. Now calls the checked WriteCountPrefix, and is renamed
   WriteUnifiedSpanPiecewise since '...Slow' described only one of its two
   entry reasons. The two header fallbacks likewise: the ask that was declined
   was larger than 23, so the smaller one may still land in place.

2. The Stingy benchmark writer capped at 64, but the largest ask in
   SET user:1 marc is 23 - so it declined nothing and measured the direct path
   twice. 16 forces the fallback but still does not fault on main; 8 does both.
   Same correction to ShortSpansStillProduceCorrectFrames, whose comment
   claimed 64 applied pressure.

3. MaxPrefixScratch was dead - every stackalloc uses MaxValueScratch or
   Sha1BulkStringLength. Deleted, wording folded into MaxValueScratch.

WriteCountPrefixSlow is now reached only from WriteCountPrefix.
- WriteUnifiedSpanPiecewise keeps NoInlining, but no longer for the reason
  documented on WriteCountPrefixSlow: there is no stackalloc left in it to
  perturb the caller's codegen. It is now simply about keeping the large-value
  body out of the hot small-value WriteUnifiedSpan frame. Said so.
- WriteHeaderSlow/WriteHeaderUnframedSlow inherited a summary saying they
  'compose in a stack buffer and let the looping write place it'. Since the
  last commit they call the checked WriteCountPrefix, which only composes if it
  is itself declined. The 'fallback when the writer declined the hint' half was
  still accurate; only the mechanism half was stale.
@mgravell
mgravell marked this pull request as ready for review September 13, 2026 13:44
@mgravell
mgravell merged commit c1a317b into main Sep 13, 2026
6 checks passed
@mgravell
mgravell deleted the marc/getspan-hint-verify branch September 13, 2026 13:44
mgravell added a commit that referenced this pull request Sep 13, 2026
…writer

Merging #3220 is behaviourally inert here: the interpolated handler owns a
pooled array and never touches IBufferWriter.GetSpan, and the MessageWriter
helpers it does call - WriteRaw(Span,...) and WriteCrlf(Span,int) - are the
span overloads, which #3220 did not change.

The underlying CLASS of defect was present, though: compute a length, then
write that many bytes without verifying the destination has room.

WriteBulk reserved 'payloadLength + HeaderMax + 2' for a $len\r\n{payload}\r\n
bulk string. HeaderMax is 12 - documented as "'*' plus up to NINE digits plus
CRLF" - but the real need is payloadLength + digits + 5, so from 1,000,000,000
bytes (10 digits) the reservation was one byte short.

It hid the same way my first #3220 test harness hid the original: ArrayPool
<byte>.Shared rounds up to a power of two, so the extra byte lands in slack.
Above 2^30 the pool switches to allocating EXACTLY the requested length, and it
bites. Verified both ways - silently absorbed at 1,000,000,000, and
IndexOutOfRangeException at 1,100,000,000.

Two distinct quantities were sharing one constant, which is how the assumption
went unnoticed: HeaderMax (the '*N\r\n' prologue) and the bulk-string prefix
happened to be the same number. Now separate, and both sized from
Format.MaxInt32TextLen rather than from a plausible digit count. That also
fixes HeaderMax itself, which was one short of the 13 a 10-digit argument count
needs - unreachable via an interpolated string, but wrong for the same reason.

Tested as arithmetic rather than by writing bytes: a write-and-check test
cannot prove this, since pool slack absorbs an under-reservation. The test
asserts the invariant at every digit-count boundary; reverting BulkReservation
to the old formula fails exactly the three 10-digit cases.

Also documented that the key-mark packing uses offset 0 as its 'no key'
sentinel, which is sound only because the prologue is reserved and the command
precedes every key - both now stated where HeaderMax is defined.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant