Do not assume GetSpan honoured the size hint - #3220
Merged
Merged
Conversation
IBufferWriter<T>.GetSpan(sizeHint) guarantees one element, not sizeHint. The "should be at least this size" wording in the docs is left over from when the parameter was called minSize; it is advisory, and consuming code is expected to test what it got and fall back. The BCL does exactly that - BuffersExtensions loops and copies in slices rather than demanding one contiguous block. See CommunityToolkit/dotnet#1208. MessageWriter treated it as a guarantee throughout: take the span, write the full amount into it, advance by that amount. The looser contract exists for transports with page limits, which can honour reasonable requests and refuse excessive ones. CycleBuffer is one - it caps the hint at 1k, sizes a fresh segment from the committed total rather than from the hint, and, the case with no floor at all, hands back a dangling recycled segment exactly as it is, whatever length that happens to be. All legitimate; the callers were wrong. It is also why this reproduces intermittently and under concurrency rather than deterministically. Reported as an ArgumentOutOfRangeException from WriteUnifiedSpan, but that is the benign manifestation and not the only site. Every fixed-size write had the same shape, and one of them is worse than an exception: var span = writer.GetSpan(expectedLength); fixed (byte* bPtr = &MemoryMarshal.GetReference(span)) Encoding.UTF8.GetBytes(cPtr, value.Length, bPtr, expectedLength); writer.Advance(expectedLength); expectedLength is passed to the encoder as the destination capacity rather than being derived from the span, so against a short span that is a buffer OVERRUN - it writes past the end of memory it does not own, silently. It now falls through to the encoder loop below it, which sizes every write from span.Length and already copes with whatever it is given. Every other site checks the length inline and hands the rare case to a separate NoInlining method that composes in a stack buffer and writes through BuffersExtensions.Write. That split is load-bearing, not style: a stackalloc in the caller's cold branch changes codegen for the whole method, and writing the fallbacks inline cost 20% on the simplest command shape. It is the same "inline-optimized for it-fits, pathological case doesn't inline" structure the BCL uses. Measured, formatting only, SET key value: 64.66ns before against 66.64ns after, so +3.1% - small error bars, so real rather than noise. That is the honest price of checking. Larger values come out ahead (203.7ns to 184.9ns), because sharing one length-prefix helper removed duplicated work. Sites fixed: both WriteHeader overloads, both WriteMultiBulkHeader overloads, WriteUnifiedPrefixedString, WriteUnifiedPrefixedBlob, WriteUnifiedInt64, WriteUnifiedUInt64, WriteUnifiedDouble, WriteInteger, WriteUnifiedSequenceIterator, the SHA1 bulk-string path, WriteUnifiedSpan and WriteCrlf. The encoder loop was already correct; the commented-out WriteUnifiedSequence is left alone. Tests use a writer that honours at most N bytes per span and places a canary past the span it hands out, so an overrun is detected rather than landing harmlessly in slack - a plain growing buffer passes while the real writer corrupts the next thing in the pool. 13 tests across the value shapes; 6 fail without the fix, with two distinct crash modes. Full suite against the docker topology: 6172 passed, 150 skipped, 1 failed - FailoverTests.SubscriptionsSurvivePrimarySwitchAsync, which fails identically on main.
Restores the NoInlining split that was lost to a git checkout during benchmarking, plus the review findings: - WriteUnifiedInt64 passed MaxPrefixScratch (30) while requesting 27; now MaxValueScratch, which is what its doc comment says it is for. - PrefixScratch slices both spans to length, so it bounds-checks in Release rather than relying on a Debug.Assert that is gone there. - WriteCountPrefix is sized for an int64: the multi-bulk header, the blob prefix and the sequence iterator all format a long and previously asked for int32 width, so the direct path could be handed exactly 14 bytes and write 23. - WriteCrlf keeps its hint of 2 rather than falling to BuffersExtensions.Write, which asks with no hint at all and would split a CRLF across segments. - The test harness checks its canary on hand-out and on reading the result, not only on Advance, so a span written past and then abandoned cannot slip through. - Coverage for WriteSha1AsHex (the tightest fit in the file), keyspace prefixes, the prefixed multi-bulk header, WriteInteger, and a long-width count.
Adds a stingy writer to the benchmark, so the fallback branch is actually exercised: the Reusable writer always returns the remainder of a 16KB buffer and so only ever measures what the CHECK costs, which the previous table could not be read as the cost of the fix generally. Corrects the numbers, which were overstated in two directions: - The -9.2% on LargeValue was partly measurement drift. main's own figure moved between 187ns and 204ns across sessions, so the claim that sharing a length-prefix helper made large values faster is not supported and is dropped. - MixedValueShapes swung between -0.4% and +4.5% across runs. BenchmarkDotNet's error bars measure within-run precision, not between-run reproducibility, and at this scale the latter is several percent. Only the SET key value figure is reproducible: 64.48ns against 66.42ns, so about +3%, measured twice independently with matching filter sets. Note the filter set matters - including the fallback benchmark in the same run moved the figure to 70.60ns, so before-and-after runs must measure the same set.
- add ShortSpansStillProduceCorrectFramesForLargeBinaryValues: byte[] takes a different route from string through WriteUnifiedSpan, and the value-shapes test only reached it with four bytes. 400/512/513 at caps of 8/400/520. - note that [InlineData(520, 512)] on the string theory does not stress that path - a 512-byte string asks for exactly 512, which a 520-cap writer honours, so it passes unfixed. Kept as regression cover, not as evidence. - relabel CountPrefixIsSizedForALong as defensive: main's long overload really does hint int32 width, but argument counts come from array lengths, so no caller can reach it. - document that TryGetSpan returning false is not side-effect free, and the two ways StingyWriter is stricter than a real writer.
519/512 is the case that bites: the hint carries int32-text slack it rarely uses, so a cap has to fall below the real frame length (1 + digits + 2 + len + 2 = 520 here), not merely below the hint, to overrun. Marked which cases fail unfixed and which are boundary cover.
…ter stingy Three findings from review: 1. WriteUnifiedSpanSlow was the NORMAL path for values over MaxQuickSpanSize, not a fallback, and called WriteCountPrefixSlow unconditionally - so every 4KB byte[] composed its $4096\r\n into a stack buffer and handed it to a hintless Write, costing a copy and risking the prefix splitting across segments: the same fragmentation just fixed in WriteCrlf. The string path never had this. Now calls the checked WriteCountPrefix, and is renamed WriteUnifiedSpanPiecewise since '...Slow' described only one of its two entry reasons. The two header fallbacks likewise: the ask that was declined was larger than 23, so the smaller one may still land in place. 2. The Stingy benchmark writer capped at 64, but the largest ask in SET user:1 marc is 23 - so it declined nothing and measured the direct path twice. 16 forces the fallback but still does not fault on main; 8 does both. Same correction to ShortSpansStillProduceCorrectFrames, whose comment claimed 64 applied pressure. 3. MaxPrefixScratch was dead - every stackalloc uses MaxValueScratch or Sha1BulkStringLength. Deleted, wording folded into MaxValueScratch. WriteCountPrefixSlow is now reached only from WriteCountPrefix.
- WriteUnifiedSpanPiecewise keeps NoInlining, but no longer for the reason documented on WriteCountPrefixSlow: there is no stackalloc left in it to perturb the caller's codegen. It is now simply about keeping the large-value body out of the hot small-value WriteUnifiedSpan frame. Said so. - WriteHeaderSlow/WriteHeaderUnframedSlow inherited a summary saying they 'compose in a stack buffer and let the looping write place it'. Since the last commit they call the checked WriteCountPrefix, which only composes if it is itself declined. The 'fallback when the writer declined the hint' half was still accurate; only the mechanism half was stale.
mgravell
marked this pull request as ready for review
September 13, 2026 13:44
mgravell
added a commit
that referenced
this pull request
Sep 13, 2026
…writer Merging #3220 is behaviourally inert here: the interpolated handler owns a pooled array and never touches IBufferWriter.GetSpan, and the MessageWriter helpers it does call - WriteRaw(Span,...) and WriteCrlf(Span,int) - are the span overloads, which #3220 did not change. The underlying CLASS of defect was present, though: compute a length, then write that many bytes without verifying the destination has room. WriteBulk reserved 'payloadLength + HeaderMax + 2' for a $len\r\n{payload}\r\n bulk string. HeaderMax is 12 - documented as "'*' plus up to NINE digits plus CRLF" - but the real need is payloadLength + digits + 5, so from 1,000,000,000 bytes (10 digits) the reservation was one byte short. It hid the same way my first #3220 test harness hid the original: ArrayPool <byte>.Shared rounds up to a power of two, so the extra byte lands in slack. Above 2^30 the pool switches to allocating EXACTLY the requested length, and it bites. Verified both ways - silently absorbed at 1,000,000,000, and IndexOutOfRangeException at 1,100,000,000. Two distinct quantities were sharing one constant, which is how the assumption went unnoticed: HeaderMax (the '*N\r\n' prologue) and the bulk-string prefix happened to be the same number. Now separate, and both sized from Format.MaxInt32TextLen rather than from a plausible digit count. That also fixes HeaderMax itself, which was one short of the 13 a 10-digit argument count needs - unreachable via an interpolated string, but wrong for the same reason. Tested as arithmetic rather than by writing bytes: a write-and-check test cannot prove this, since pool slack absorbs an under-reservation. The test asserts the invariant at every digit-count boundary; reverting BulkReservation to the old formula fails exactly the three 10-digit cases. Also documented that the key-mark packing uses offset 0 as its 'no key' sentinel, which is sound only because the prologue is reserved and the command precedes every key - both now stated where HeaderMax is defined.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the intermittent
ArgumentOutOfRangeExceptionfromMessageWriter.WriteUnifiedSpanreported inthis gist — and, while there, a
silent buffer overrun on a neighbouring path.
The bug
IBufferWriter<T>.GetSpan(sizeHint)guarantees one element, notsizeHint. The "should be at leastthis size" wording in the docs is left over from when the parameter was called
minSize; it is advisory,and consuming code is expected to test what it got and fall back — which is exactly what the BCL's own
BuffersExtensionsdoes, looping and copying in slices rather than demanding one contiguous block. SeeCommunityToolkit/dotnet#1208.
MessageWritertreated it as a guarantee throughout: take the span, write the full amount into it,advance by that amount.
The looser contract exists for transports with page limits, which can honour reasonable requests and
refuse excessive ones.
CycleBufferis one — it caps the hint at 1k, sizes a fresh segment from thecommitted total rather than from the hint, and, the case with no floor at all, hands back a dangling
recycled segment exactly as it is, whatever length that happens to be:
All legitimate. The callers were wrong.
That intermittency is specific to
CycleBuffer— the recycled-spare path is what makes the reportedfailure show up under concurrency rather than every time. It is worth separating out, because the other
under-delivering writer is not intermittent at all.
BlockBuffer.GetBufferclamps the hint to[16, 128]and then returns whatever is left in the block, and says so itself:
So on
BlockBufferthis is a straightforward capacity bug: any request above 128 is under-served wheneverthe block has between 128 and 527 bytes left, which covers the whole quick-span path.
BlockBufferUnderDeliversWithoutAnyContrivedWriterexercises it throughTestHarness.Writewith no synthetic writerinvolved at all.
On
main, that test does not throw — the test host dies with exit code 139 (SIGSEGV), reproducedtwice. That is the "silent overrun" characterisation, executed rather than reasoned.
The reported exception is the benign case
Every fixed-size write had the same shape, and one of them is worse than an exception:
expectedLengthis passed to the encoder as the destination capacity rather than being derived from thespan, so against a short span that writes past the end of memory it does not own — silently. It now
falls through to the encoder loop immediately below it, which sizes every write from
span.Lengthandalready copes with whatever it is given.
Shape of the fix
The hot sites check the length inline and hand the rare case to a separate
NoInliningmethod thatcomposes in a stack buffer and writes through
BuffersExtensions.Write.That split is load-bearing rather than stylistic. A
stackallocin the caller's cold branch changescodegen for the whole method: writing the fallbacks inline cost 20% on the simplest command shape. It
is the same "inline-optimized for it-fits, pathological case doesn't inline" structure the BCL uses, and
the issue above describes. The genuinely cold sites (
WriteUnifiedUInt64,WriteUnifiedDouble,WriteInteger, the SHA1 path) keep aPrefixScratchhelper, which slices both spans to the requestedlength so it bounds-checks in Release rather than relying on a
Debug.Assert.Sites: both
WriteHeaderoverloads, bothWriteMultiBulkHeaderoverloads,WriteUnifiedPrefixedString,WriteUnifiedPrefixedBlob,WriteUnifiedInt64,WriteUnifiedUInt64,WriteUnifiedDouble,WriteInteger,WriteUnifiedSequenceIterator, the SHA1 bulk-string path,WriteUnifiedSpanandWriteCrlf. The encoder loop was already correct; the commented-outWriteUnifiedSequenceis untouched.Also fixed while in there
int64.WriteMultiBulkHeader(long),WriteUnifiedPrefixedBlob(
prefix.LongLength + value.LongLength) andWriteUnifiedSequenceIteratorall format alongbut askedfor
3 + MaxInt32TextLen= 14, against a possible 23. The commented-outWriteUnifiedSequencehad itright. Now covered by a test — but defensively, not as a live bug: the test fails on
mainonlybecause it calls
WriteMultiBulkHeader(long.MaxValue)directly, and no caller can reach that, sinceargument counts come from array lengths. The sizing follows the parameter type rather than today's
callers; the test comment says the same.
WriteUnifiedSpansends everything overMaxQuickSpanSize(512) down the piecewise path because no single span could hold it — so for a 4KBbyte[]that path is the normal one, not a fallback, yet it calledWriteCountPrefixSlowunconditionally. That composed
$4096\r\ninto a stack buffer and handed it to a hintlessBuffersExtensions.Write: an extra copy, and the same length-prefix-split-across-segmentsfragmentation this PR fixes two hunks earlier in
WriteCrlf. The string path never had it(
WriteUnifiedPrefixedStringcalls the checkedWriteCountPrefix), so it was asymmetric for no reason.It now calls
WriteCountPrefixtoo, and is renamedWriteUnifiedSpanPiecewise—…Slowdescribed onlyone of its two entry reasons. The two header fallbacks got the same treatment: the ask that was declined
was larger than 23, so the smaller one may well still land in place.
WriteCountPrefixSlowis nowreached only from
WriteCountPrefix.WriteUnifiedInt64passed the wrong scratch constant -MaxPrefixScratch(30) while requesting 27.It fit by three bytes, guarded only by a
Debug.Assertthat is gone in Release.WriteCrlfkeeps its hint of 2. Falling through toBuffersExtensions.Writewould ask with no hintat all, and
CycleBuffer'shint <= 0branch returns anything non-empty - so a 1-byte segment tailwould split a CRLF across segments rather than rolling to a fresh one. Protocol-correct either way, but
a fragmentation change on a per-bulk-string path is not one to make by accident.
Cost
Formatting only, no server, via the added
MessageWriterBenchmarks:SET key valueThat is the honest price of checking, and it is reproducible — measured twice independently, at +3.1% and
+3.0%.
Only that figure is claimed. The mixed-value and 4KB cases swung by several percent between runs in
both directions (
main's own 4KB figure moved between 187ns and 204ns across sessions), so they are notreported: BenchmarkDotNet's error bars measure within-run precision, not between-run reproducibility. The
filter set also matters — including the fallback benchmark in the same run moved
SET key valueto70.6ns, so before-and-after runs must measure the same set.
The benchmark has two writers, because they answer different questions.
Reusablealways returns theremainder of a 16KB buffer, so the hint is always honoured and the table above measures what the check
costs.
Stingynever returns more than 8 bytes, so every burst takes the fallback; that one cannot becompared against a pre-fix build, because a pre-fix build faults.
8 is deliberate, and an earlier 64 was useless: the largest ask in
SET user:1 marcis 23 (WriteHeaderwants 9 framed command bytes + 3 +
MaxInt32TextLen;WriteCountPrefixwants 3 +MaxInt64TextLen), so a64-byte cap declines nothing and measures the direct path twice. 16 declines the two 23-byte asks but still
does not fault on
main, whoseWriteHeaderwrites only 13 of the 23 it asked for. 8 exercises everyfallback and faults on
main— confirmed by runningShortSpansStillProduceCorrectFrames(max: 8), thesame command shape, against
main.Tests
BufferWriterHintTestsuses a writer that honours at most N bytes per span and places a canary past thespan it hands out, verified on
Advance, on the next hand-out, and on reading the result — all three,because a span written past and then abandoned without advancing would slip through a check on
Advancealone.
That detail matters: my first version of this harness used a plain growing buffer and passed against the
unfixed code — an overrun lands harmlessly in the slack that follows, while the real writer corrupts the
next thing in the pool.
Direct coverage for the paths no message shape reaches:
WriteSha1AsHex(the tightest fit in the file -47 requested, exactly 47 written, no slack), keyspace-isolation prefixes, the prefixed multi-bulk header,
WriteInteger, and a long-width count.34 tests. Against
main, 16 of the 31 synthetic-writer tests fail — and the 3BlockBuffertests cannotbe counted alongside them, because they take the test host down with SIGSEGV before the run finishes.
Verification
Full suite against the docker topology: 6184 passed, 150 skipped, 1 failed. The failure is
FailoverTests.SubscriptionsSurvivePrimarySwitchAsync, which fails identically onmain.CommandTimeoutTests.DefaultHeartbeatLowTimeoutandDatabaseTests.CountKeysappeared in one run, wereabsent from the next, and pass consistently in isolation — flaky, not related. Full
dotnet build Build.csproj -c Release /p:CI=truepasses.Not addressed
CycleBufferis unchanged, deliberately — it is within its rights, and the callers were the problem.Separately noticed while investigating:
ScriptReadOnlyCommandTests,ScriptEvalRespNoScriptTestsandRenderedArgsLeakTestsfail rather than skip when no server is present, because they hand-roll theirconnection to get a command-map hook (
TestBase.Createhas none) and so lose theAbortOnConnectFail = falseplusAssert.SkipUnless(conn.IsConnected, ...)thatTestBaseapplies. Nottouched here, but it reports "broken" instead of "skipped" on any machine without Redis, which is a
misleading signal when you are trying to tell whether your own change broke something.