Skip to content

Let the base58 cancellation budget span a whole block of values - #119388

Open
groeneai wants to merge 4 commits into
ClickHouse:masterfrom
groeneai:groeneai/base58-cancellation-across-rows
Open

Let the base58 cancellation budget span a whole block of values#119388
groeneai wants to merge 4 commits into
ClickHouse:masterfrom
groeneai:groeneai/base58-cancellation-across-rows

Conversation

@groeneai

@groeneai groeneai commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Changelog category (leave one):

  • Bug Fix (user-visible misbehavior in an official stable release)

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Fixed max_execution_time and KILL QUERY being ignored while base58Encode or base58Decode converts a block of values that are each too small to reach the cancellation checkpoint. One 1025-byte conversion accumulates about 70% of the work between two checks, so a block of 65,536 of them ran uninterrupted for minutes. base58Encode of exactly 32 or 64 bytes uses a fixed-size encoder and is unchanged.

Description

Requested by @ alexey-milovidov in #118222 (comment). His report: an AST fuzzer mutation of 04027_base58_optimized.sql over 1025-byte values, is_cancelled: 1, elapsed: 762 s.

Root cause. work_since_check in encodeBase58 and decodeBase58 was call-local, so the budget restarted at every value. Encoding n bytes costs 0.683 * n^2 units against work_per_check = 1 << 20, so the first checkpoint arrives near 1239 bytes: a 1025-byte value gets zero checks and the count dies with it, leaving that 65,509-row block uninterruptible. function_base58_max_input_size bounds one value, never the block, and 1025 bytes is well inside its 10 KB default.

The change. Both conversions take a defaulted size_t * shared_work_since_check, bound as a reference so every exit path stays charged. FunctionBaseXXConversion::executeImpl owns one counter and threads it through its four row loops. Callback, interval and per-value cadence are unchanged: GenericCancellationInterval still reports exactly 63 calls.

Validation. Debug, 65536 rows, server elapsed, 50 ms deadline: encode 2097.1 ms to 50.6 ms, decode 509.9 ms to 50.5 ms. Two unit tests: eight sub-threshold values reach 0 checks with a per-call count and 5 with a shared one (encode, decode, and values rejected at their last character), and the two row loops, driven directly, make the same 5, or 0 if a row gets its own count. Only executeImpl's own counter rests on those numbers. tests/performance/base58.xml is unchanged.

Disclosed, not fixed here. base58Encode of exactly 32 or 64 bytes takes a fixed-size encoder with no checkpoint, so such a block keeps ordinary per-block granularity: 10M rows notice a deadline 0.6 s (32 B) or 1.8 s (64 B) late on debug, and SHA256 2.1 s late on the same block. With function_base58_max_input_size = 0, an all-zero or all-'1' value of hundreds of megabytes runs an O(n) prologue with no checkpoint. Both are absent checks, not wrong-lifetime counts.

Related: #112203


Workflow [PR]
Sync PR [sync-upstream/pr/119388]

groeneai and others added 4 commits September 10, 2026 23:52
The generic base58 encoder and decoder invoke their cancellation callback once
they have accumulated 1 MiB of inner-loop work. That counter was call-local, so
it restarted at every value: with this tree's constants a value only reaches the
budget at about 1239 bytes to encode or 1692 characters to decode, and the count
was thrown away when the value ended. A block of values below that size therefore
never checked for cancellation at all, and the first cancellation point was the
pipeline's own check after the whole block.

That is how a cancelled query stayed inside one function evaluation for 762
seconds in the amd_tsan stress test: 65509 rows of 1025 bytes, each of them
accumulating only about 70% of the work between two checks.

Give the caller ownership of the counter instead, so one budget spans every value
of the block, which is the lifetime of the executeImpl call that owns it. The
callback interface, the 1 MiB threshold and the per-value cadence for a single
large value are all unchanged; a caller that passes no counter gets exactly
today's per-call behaviour. The linear base32 and base64 conversions only gain an
ignored parameter: they cannot run long enough to need a checkpoint.

Binding a reference rather than copying the count in and out is what keeps every
exit path correct, including the decoder's early return on an invalid character:
work done before a value is rejected stays charged, which is what a block of
invalid values passed to tryBase58Decode depends on.

Related: ClickHouse#112203

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A 1025-byte conversion accumulates about 0.68 of the work between two checks, so
with a count of its own it reaches none: eight such values in a row invoke the
callback zero times. The test pins that, then pins the five calls the same eight
values make when the caller owns the count, for encode, for decode, and for
values rejected at their last character, where the work is charged from inside
the loop that then returns empty.

The two zero arms are what make the fives meaningful. They rule out one value
being large enough to check on its own, so the test cannot be satisfied by
shrinking the interval instead of keeping the count alive across values.

It is a unit test rather than a SQL one because both a fixed and an unfixed build
raise the same TIMEOUT_EXCEEDED with the same message text for such a block, and
only the elapsed value differs, so an SQL oracle would have to be a wall-clock
bound. The uninterruptible stretch here is memory-bound at about 10 ms per MB of
block, which puts a two-sided margin at roughly 620 MB, and timing-prone
cancellation tests of that shape were removed from master in d5bc955.

The row-loop threading through executeImpl is not covered here, since the
conversions are called directly. It stays covered by review and by the
end-to-end measurement quoted in the pull request description.

Related: ClickHouse#112203

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The existing arm calls encodeBase58 and decodeBase58 directly and keeps the count
itself, so it pins the conversions but not the row loops above them, which are
where a query's block is converted. Those loops are public static functions whose
cancellation callback is a plain std::function, so one can be driven from a unit
test with no clock, no server and no query context: build a column, call the loop
once, count the callbacks.

Eight rows of 1025 bytes invoke the callback five times in each direction, the
same five the same eight values make at the primitive level. Making either loop
give each row a fresh count, which is the behaviour before the count moved to the
caller, drops that to zero and reds this test while all ten of the primitive
tests stay green.

This supersedes the closing note of the previous commit: the loops that convert a
block of String or FixedString values are covered here, and the measurement
quoted in the pull request description now only stands in for the single counter
executeImpl declares, whose scope no unit test can enter, because the callback
does not exist without a query context. Passing the count by value there is not
observable either way: executeImpl calls one loop once per block, so a copy still
spans the block and its value after the call is unused.

Related: ClickHouse#112203

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The header paragraph says in three lines what it said in four, and the note in
encodeBase58 drops a sentence about the reference keeping every exit path
correct: that function has no early return below the binding, so the sentence
described the decoder instead, where it is kept.

Code is unchanged: stripping comments from both revisions of either file leaves
identical text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@groeneai groeneai added can be tested Allows running workflows for external contributors groeneai-origin-request PR origin: a maintainer pinged or directed groeneai labels Sep 11, 2026
@groeneai

Copy link
Copy Markdown
Collaborator Author
Internal second-model review: adjudication log (click to expand)

Pre-publication review by an independent model (engine: codex; 3 findings over 2 passes, plus 5 from my
own cold passes), bounded at 1 full pass plus one fix round with a delta recheck.

# Sev Finding Verdict Evidence / action
1 The shared count ignores the leading-zero and leading-'1' prologue, so a block of zero-like values stays uninterruptible (src/Common/Base58.cpp) DISAGREE The prologue does escape the count, but charging it fixes nothing: check_cancellation() is called at exactly two sites, Base58.cpp:692 and :858, both inside the pass loop that a zero-like value never enters, so interrupting such a block needs a new checkpoint on the prefix path. That is an absent check rather than a count with the wrong lifetime, and its cost is linear in the bytes read, the shape this file's own untouched comments leave callback-free for base32 and base64. Disclosed in the description; at the default 10 KB limit the block variant needs no setting change, which the description's single-value sentence does not spell out
2 The budget still resets per internal type when the argument is Dynamic: the adaptor runs a fresh nested execution per alternative, and FunctionDynamicAdaptor polls no cancellation state at all, unlike FunctionVariantAdaptor.cpp:497 DISAGREE Mechanism conceded in full, consequence refuted. The adaptor partitions by type NAME, never by row (FunctionDynamicAdaptor.cpp:428-435, :449-457), so every String row of a block lands in one partition and shares one budget: the escape cannot scale with row count, which is the entire defect (65,509 rows in the report). It scales instead with the number of distinct types whose partition total stays under one interval, i.e. String plus FixedString(N) below the per-value threshold, since a wider FixedString crosses an interval on a single value and breaks the chain. Re-deriving from the measured w(1025) = 715,760 units gives 0.68127 units/byte^2, a threshold of 1241 bytes and one interval at 0.0469 ms on this debug build, so the worst-case uninterruptible stretch reachable through Dynamic is about 58 ms, against the 2097 ms this PR removes from the same block shape and the 762 s in the report. That is 19x the cadence this PR deliberately preserves (one 10 KB value at the default limit is 68 intervals, 3.0 ms un-checked), not a return of the defect. The adaptor gap itself is real, and it is the adaptor's for every expensive function routed through it, so the fix belongs in an adaptor-scoped change: that is what #113612 did for the Variant sibling. Folding it here would put a generic-adaptor change behind a base58 changelog entry and cover two root causes
3 ⚠️ The regression test proves the primitive shares a caller's count, but stays green if the row loops stop sharing it AGREE, fixed @ a156c91 Added one deterministic unit test driving BaseXXEncode::processFixedString and BaseXXDecode::processString over a block of sub-threshold rows, asserting exact call counts. Giving either loop a fresh count per row, which is the behaviour before this change, drops them from 5 to 0 and reds the new test while all ten primitive tests stay green. Correcting my own earlier note on this: changing a loop's size_t & parameter to size_t does not red anything and no test could catch it, because executeImpl calls one loop once per block, so a copy still spans the block and its value afterwards is unused. The counts were read off a deliberately wrong literal first, so they are measured rather than tuned
4 💡 Two comment and attribution nits in the description and headers AGREE, fixed @ 688a6da The 1 << 20 interval is #106428's; the exact-count assertion is from #117619. Description corrected and two comments condensed

I did not adopt CancellationBudget for the shared count: it counts iterations against 1 << 16 rather
than the 1 << 20 of work #106428 chose here, so it would move the cadence this change is preserving.

Severity: ❌ blocker / ⚠️ major / 💡 nit. DISAGREE verdicts carry recorded evidence and are terminal
per finding.

Session id: cron:clickhouse-review-slot-9:20260911-031400

@clickhouse-gh

clickhouse-gh Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Workflow [PR], commit [688a6da]

Summary:

job_name test_name status info comment
Stateless tests (amd_debug, distributed plan, s3 storage, parallel) FAIL
03100_lwu_deletes_4_index FAIL cidb

AI Review

Summary

This PR moves the generic base58 cancellation budget from a per-value counter to a block-wide counter, which fixes the 1025-byte regression described in the PR body. The remaining problem is that base58Encode still bypasses that shared budget on its specialized 32-byte and 64-byte paths, so the user-visible timeout / KILL QUERY contract is still incomplete for common key/hash inputs.

Findings

⚠️ Majors

  • [src/Functions/FunctionBase58Conversion.h:39-42] base58Encode promises block-wide cancellation accounting here, but the 32-byte and 64-byte fast paths return encodeBase58_32 / encodeBase58_64 directly and never touch work_since_check or check_cancellation. FunctionBaseXXConversion has no outer checkpoint in its row loops, so a block of these values still runs as one uninterruptible stretch and can exceed max_execution_time or ignore KILL QUERY until the whole block finishes.
    Suggested fix: either make the fixed-size encoders charge/poll too, or add a row-loop-level checkpoint that also covers these branches.
Tests
  • ⚠️ The new row-loop regression test deliberately keeps the input on the generic path (1025 bytes), so it never exercises the specialized 32-byte / 64-byte encoder that still bypasses cancellation. A focused unit test for those two sizes would prove the full base58Encode contract.
Final Verdict

Changes requested: the advertised base58Encode cancellation fix is still incomplete for the specialized 32-byte and 64-byte encoder paths.

LLVM Coverage Report

Measured on commit 688a6da.

Metric Baseline Current Δ
Lines 89.00% 89.00% +0.00%
Functions 91.80% 91.80% +0.00%
Branches 81.30% 81.30% +0.00%

Changed lines: Changed C/C++ lines covered: 129/134 (96.27%) · Uncovered code

Full report · Diff report

@clickhouse-gh clickhouse-gh Bot added the pr-bugfix Pull request with bugfix, not backported by default label Sep 11, 2026
@groeneai

Copy link
Copy Markdown
Collaborator Author

cc @rschu1ze @nickitat, could you review this? encodeBase58 and decodeBase58 kept their cancellation budget in a call-local counter, so it restarted at every value, and a block of values each below the checkpoint threshold never polled max_execution_time or KILL QUERY at all; the caller now owns one counter for the whole block. The regression tests are unit tests rather than SQL because a fixed and an unfixed server raise the same TIMEOUT_EXCEEDED with the same message text for such a block and only the elapsed value differs, so an SQL oracle would have to be a wall-clock bound.

Comment on lines 39 to 42
if (src.size() == 32)
return encodeBase58_32(reinterpret_cast<const UInt8 *>(src.data()), dst);
else if (src.size() == 64)
return encodeBase58_64(reinterpret_cast<const UInt8 *>(src.data()), dst);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

base58Encode still leaves the two most common fast paths outside the new block-wide cancellation accounting. For src.size() == 32 or 64 this branch returns encodeBase58_32 / encodeBase58_64 directly, so work_since_check is never charged and check_cancellation is never polled; processString / processFixedString do not have any outer checkpoint of their own. That means a block of 32-byte or 64-byte values can still ignore max_execution_time and KILL QUERY until the whole block finishes, which is the same user-visible contract this PR says it fixes for base58Encode.

Please either make the fixed-size encoders charge/poll too, or add a row-loop-level checkpoint that also covers these branches.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified, and the mechanism is exactly as described: Base58EncodeTraits::perform routes src.size() == 32 and == 64 to encodeBase58_32 / encodeBase58_64, which take neither the callback nor the counter, and the row loops carry no checkpoint of their own.

Measured on this branch (debug, server-reported elapsed at the throw, one block of 10M values, 400 ms deadline): 32 B is noticed 0.63 s late, 64 B 1.81 s late, 33 B and 65 B 1 ms late. On the same block SHA256 is 2.10 s late and hex 0.08 to 0.15 s late, and neither of those polls at all. At the default max_block_size the fixed-size stretch is 4.4 ms (32 B) and 10.1 ms (64 B), against the 2097 ms this PR removes from a 65,536-row block of 1025-byte values.

So the escape is real, but it is the ordinary per-block granularity and not this PR's defect: per-row cost in those two encoders is a constant, and in this tree an intra-block checkpoint exists only where per-row work grows with the value (the CancellationBudget users, the h3 expansions, the generic base58 conversion). The row-loop checkpoint suggested as the alternative sits in the loop that also serves base64Encode and base32Encode, whose traits deliberately ignore the callback, so it would be a framework change owed equally to them and to hex, not a base58 fix. #106428, which introduced base58 cancellation, was written with this dispatch already in place and passed the callback only to the generic conversion, so this PR leaves that cadence as it found it.

The point about the claim stands and I have acted on it: the changelog entry now ends with "base58Encode of exactly 32 or 64 bytes uses a fixed-size encoder and is unchanged", and the description discloses the branch with the numbers above.

@clickhouse-gh clickhouse-gh Bot added the comp-regular-function Regular scalar functions: string processing, data conversion, arithmetic, math, comparison, condi... label Sep 11, 2026
@clickhouse-gh

clickhouse-gh Bot commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

Build profile diff (arm_release)

Comparing 688a6da2e with master c108e273b (stripped binary size, per-symbol sizes and ThinLTO time; compile times per translation unit against the most recent warmup build that recompiled it).

✅ No significant changes.

Binary sizes

programs/clickhouse-stripped: smaller than the master baseline by the known offset between the two builds, so the difference is not shown. A delta that differs from the offset by more than 50% of it is shown, in either direction.

The official master build is compiled with -g and a pull request build is not, and XRay counts debug instructions towards its instrumentation threshold, so master instruments thousands of functions more and its binary is ~0.4% larger no matter what the pull request does.

Compile time of recompiled translation units

19 translation units recompiled, 55 s compile time in total, 19 of them have a recent master baseline.

Job report

@groeneai

Copy link
Copy Markdown
Collaborator Author

CI finish ledger - 688a6da

Every failure below has an owner: a fixing PR (mine or external), or a full-effort fix task
whose fixing-PR link will be posted here when it opens. Only CH Inc sync is exempt.

Check / test Reason Owner / fixing PR
Stateless tests (amd_debug, distributed plan, s3 storage, parallel) / 03100_lwu_deletes_4_index trunk defect, first seen on true master at 592da49: the second lightweight DELETE reports read_rows = 0 instead of 8 because a read step that materializes no column from the part reports no rows read #119400 (ours, open)
Mergeable Check, PR praktika aggregators rolling up the row above, not separate failures #119400 (ours, open)

Not caused by this pull request. This branch only makes the base58 encoder and decoder
interruptible; 03100_lwu_deletes_4_index fails the same way on true master, where the runner's own
randomized-settings diagnosis reproduces it, and on branches that do not carry it.

Session id: cron:our-pr-ci-monitor:20260911-093124

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

can be tested Allows running workflows for external contributors comp-regular-function Regular scalar functions: string processing, data conversion, arithmetic, math, comparison, condi... groeneai-origin-request PR origin: a maintainer pinged or directed groeneai pr-bugfix Pull request with bugfix, not backported by default

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant