Add ALTER TABLE ... RECOMPRESS COLUMN - #109453

Preview deployment for your docs. Learn more about Mintlify Previews.

Workflow [PR], commit [0c9491a]

Summary: ✅

• Performance Comparison: Performance dashboard

AI Review

Summary

This PR adds ALTER TABLE ... RECOMPRESS COLUMN and does a substantial amount of correctness work around lossy codecs, dependent projections / indices / TTLs, shared Nested offsets, access control, and parser / formatter round-trips. I found one remaining blocker: the new mutation verb is persisted into the existing text mutation format without any compatibility gate, so mixed-version replicated clusters can produce mutation entries that older replicas cannot parse.

Findings

❌ Blockers

• [src/Storages/MutationCommands.cpp:203] RECOMPRESS COLUMN is now serialized as a normal mutation command, but replicated and local mutation entries still use the existing text format version: 1 and are replayed through ParserAlterCommandList. That means a newer server can persist RECOMPRESS COLUMN ... into /mutations or mutation_*.txt, and an older binary will fail to load or execute that pending mutation during a rolling upgrade or downgrade/restart. The feature needs a compatibility gate: either reject it while older replicas may still exist, or version the persisted mutation representation and provide an encoding older binaries can survive.

Final Verdict

Status: ❌ Block
Minimum required action: make persisted RECOMPRESS COLUMN mutations safe for mixed-version replicated clusters and pending local mutation replay, either by gating the feature until all replicas support it or by versioning the mutation serialization format.

LLVM Coverage Report

Measured on commit 0c9491a.

Changed lines: Changed C/C++ lines covered: 591/640 (92.34%) · Uncovered code

Full report · Diff report

Addressed the AI-review findings and the stress-test crash (pushed in 7ff895bd9ed and 537c5058954):

1. Heap-buffer-overflow in the wide-part fast path (the cause of the amd_asan_ubsan stress failure — 12 identical ASan aborts in RawCompressedBlockReader::readBlock). The decompression buffer was sized to exactly the decompressed size, but LZ4 wildCopyFromInput overruns by up to getAdditionalSizeAtTheEndOfBuffer bytes. Now reserves that trailing slack (mirroring CompressedReadBuffer::nextImpl) while keeping the logical size at the exact decompressed length.

2. Inherited default codec (MergeTreeColumnRecompression.cpp finding). A column without an explicit CODEC(...) was recompressed with the part's stored default_codec, so RECOMPRESS COLUMN did nothing after a default_compression_codec change. Such columns are now routed through the whole-part rewrite, which re-serializes with the current effective codec and rewrites default_compression_codec.txt.

3. UNIQUE KEY safety (MutateTask.cpp finding). RECOMPRESS COLUMN is now rejected in checkMutationIsPossible on tables with a UNIQUE KEY, because the whole-part rewrite does not preserve the delete_bitmap_*.rbm sidecars.

Regression coverage: 04402_recompress_column now also exercises an LZ4-compressed source (the overflow path) and an inherited-default column after a default_compression_codec change; new 04403_recompress_column_unique_key_guard covers the UNIQUE KEY rejection.

The arm_release stress failure (Logical error: Unexpected return type ... in decorrelateQueryPlan) is unrelated — a known AST-fuzzer-found correlated-subquery decorrelation issue (#107445, #106377, #107951); this PR touches no planner/decorrelation code.

Addressed the remaining AI-review blocker (default-dependent explicit codecs) in aeda730a0b9.

The recompress_needs_full_rewrite gate only checked !column_desc->codec, so a column with an explicit codec AST that references Default — either directly (CODEC(Default)) or inside a pipeline (CODEC(Delta, Default)) — still took the wide in-place fast path. That path resolves the codec against the part's stored default_codec, which is not updated on a default_compression_codec change, so RECOMPRESS COLUMN was a no-op on wide parts for those columns.

Since Default resolves through the current_default argument of CompressionCodecFactory::get exactly like an inherited-default column, the new helper codecDependsOnDefault (which mirrors how the factory parses the codec AST) now routes those columns through the whole-part rewrite as well. That path re-serializes with the current effective codec (getCompressionCodecForPart) and rewrites default_compression_codec.txt.

Regression coverage: new 04506_recompress_column_default_codec_ast exercises CODEC(Default) and CODEC(Delta, Default) on wide parts after a default_compression_codec change (verified locally: the column shrinks from a NONE-sized to a ZSTD-sized column, data intact, CHECK TABLE passes).

Addressed the remaining AI-review blocker (Nested shared-offset columns) in b2229aba24b.

RECOMPRESS COLUMN recompresses one column at a time via recompressColumnStreams. With share_nested_offsets enabled, Nested siblings such as n.a and n.b share a single on-disk offsets stream (n.size0), so ALTER ... RECOMPRESS COLUMN n.a, RECOMPRESS COLUMN n.b reached that shared .bin/marks pair once per sibling and rewrote it twice, with the last pass deciding the codec of the shared stream.

The data itself stayed readable — the compression codec is self-describing via the block header, and each pass always reads the pristine source stream and writes a self-consistent block/marks pair, so CHECK TABLE passes — but the shared stream was written more than once and its codec was non-deterministic. The fix shares a recompressed_streams set across the per-column calls so every on-disk stream is rewritten exactly once, by the first column that reaches it. This mirrors the wide-part writer, which also writes a shared offsets stream only once.

Regression coverage: new 04600_recompress_column_nested_shared_offsets recompresses two Nested siblings with different codecs (and, separately, only one sibling) on wide parts, and validates the data, the rewritten marks (point lookup + scattered scan), and CHECK TABLE.

Addressed the remaining AI-review blocker (non-deterministic codec for the shared Nested offsets stream) in ba5706ed928.

The earlier recompressed_streams dedup already rewrote the shared n.size0 stream exactly once, but the sibling whose codec won was taken from columns_to_recompress, which is a NameSet (std::unordered_set). Its iteration order is hash-bucket order, so for siblings with different codecs the shared stream received an arbitrary codec from run to run — neither the ALTER query order nor any stable schema order.

Fix: ctx->columns_to_recompress is now built by iterating the part's stored-column order and filtering by the recompress set, so the first sibling in stored order deterministically owns the shared stream. This matches how a fresh write of the part assigns the shared offsets stream (the first sibling in stored order creates it), so RECOMPRESS COLUMN and a from-scratch write agree.

New test 04601_recompress_column_shared_offsets_codec inspects the raw compression method byte of the shared n.size0.bin and asserts it takes the schema-first sibling's codec (ZSTD, 0x90) regardless of the column order in the ALTER (it runs the ALTER with both n.a, n.b and n.b, n.a and expects the same codec both times).

Addressed the AI review Request changes verdict (non-physical RECOMPRESS COLUMN targets).

RECOMPRESS COLUMN is parsed straight into a MutationCommand and bypasses AlterCommands validation, so an unknown / ALIAS / EPHEMERAL target reached the per-part mutation path: a silent no-op on wide/full parts (the column is absent from the part), or a whole-part rewrite of unrelated physical columns on compact/non-full parts.

Fix (610205723d3): validate the target in MergeTreeData::checkMutationIsPossible, before per-part dispatch — reject an unknown name with NO_SUCH_COLUMN_IN_TABLE (16) and an ALIAS/EPHEMERAL column with BAD_ARGUMENTS (36); neither has an on-disk stream to recompress. Physical (Ordinary/Materialized) columns, including flattened Nested subcolumns like n.a, still pass through (verified 04600 unchanged; the UNIQUE KEY guard in 04403 still fires with 344 for physical columns).

New test 04602_recompress_column_non_physical_guard: asserts unknown → 16, ALIAS → 36, EPHEMERAL → 36, and a physical column succeeds.

Addressed the AI review verdict at `610205723d3`. Pushed `4f32ff3ab8e` + `e90b68d6d3a`.

Blocker 1 — MergeTreeColumnRecompression.cpp source-stream lookup (fixed). forEachColumnStream recomputed each stream's on-disk name from the table's current replace_long_file_name_to_hash / max_file_name_length, so after those settings change the recomputed name no longer matched the file on disk and the wide fast path silently skipped the stream (no-op). Now resolved against the source part's recorded files via IMergeTreeDataPart::getStreamNameForColumn (tries plain + hashed name), which also subsumes the old existence check. New test 04603_recompress_column_file_name_hash_change reproduces the no-op (column stays NONE-sized) on the old code and passes with the fix.

Major — StorageMemory silent no-op (fixed). StorageMemory::mutate dropped RECOMPRESS_COLUMN and checkMutationIsPossible did no validation, so ALTER TABLE mem RECOMPRESS COLUMN x succeeded as a no-op. Now rejected up front in StorageMemory::checkMutationIsPossible with NOT_IMPLEMENTED (also covers MaterializedView targets forwarding to Memory; a combined UPDATE …, RECOMPRESS … is rejected whole; other Memory mutations unaffected). New test 04604_recompress_column_memory_guard.

Blocker 2 — RENAME COLUMN + RECOMPRESS COLUMN (not reproducible; no change). This combination cannot reach the reported line with a mismatched name. RENAME COLUMN parses as an AlterCommand, RECOMPRESS COLUMN as a MutationCommand, so they land in separate command segments → separate mutation entries; and RENAME_COLUMN is a barrier command (isBarrierCommand), so its entry is never combined with any other into one MutateTask (StorageMergeTree/ReplicatedMergeTreeMergePredicate). The rename therefore always applies first in version order, and RECOMPRESS runs on a source_part that already knows the renamed column. Verified on a wide part, both statement orders:

• ALTER TABLE t RENAME COLUMN x TO y, RECOMPRESS COLUMN y and ALTER TABLE t RECOMPRESS COLUMN x, RENAME COLUMN x TO y: no exception, CHECK TABLE = 1, rows intact.
• With a codec change first (x CODEC(NONE) → MODIFY … CODEC(ZSTD) → RENAME x TO y, RECOMPRESS y): the column shrinks 10.8 MB → 27 KB and count(y = repeat('a',100)) = 100000, i.e. the recompression genuinely runs (not silently dropped) under the post-rename name.

Happy to add an explicit guard rejecting the combination if you'd prefer defense-in-depth, but it looks unreachable today.

Updating the branch to fix the 04001_join_reorder_through_expression test, which was broken today in master.

🕵 Pushed 81331f437503 (master merge 31c38c24665d + 7f421b409f15 + 81331f437503).

AI review Blocker (MutateTask.cpp:384) — fixed. The whole-part fallback of RECOMPRESS COLUMN was decided in splitAndModifyMutationCommands but re-derived in the task selection from MutationsInterpreter::isAffectingAllColumns, which compares the interpreter output against metadata_snapshot->getColumns().getNamesOfPhysical(). A wide part that predates an unrelated ADD COLUMN stores fewer columns than the metadata has, so the predicate was false, the mutation went through MutateSomePartColumnsTask, and the writer kept source_part->default_codec — a target inheriting the table codec (or using CODEC(Default)) was rewritten with the pre-ALTER default codec, i.e. a silent no-op. The decision is now carried explicitly from splitAndModifyMutationCommands to rewritesAllPartColumns, so both always agree.

New test 05030_recompress_column_inherited_default_after_add_column, exactly the case the review asked for. Validated both ways locally: green with the fix; with the new term neutered it prints zstd is small 0.

All 12 red stateless jobs — the same failure, 05023_recompress_column_dropped_target, fixed. Root cause: ALTER TABLE ... DROP COLUMN b was rejected by MergeTreeData::checkDropOrRenameCommandDoesntAffectInProgressMutations because the queued RECOMPRESS COLUMN b names the same column. The test only ever passed when the recompression happened to finish first — number_of_free_entries_in_pool_to_execute_mutation cannot keep a mutation queued on an idle pool (CompactionStatistics short-circuits on occupied <= 1), so this was a pure race, not the packed-part effect I assumed in the previous pass.

A pending RECOMPRESS COLUMN now no longer blocks DROP COLUMN of its target: the recompression only re-serializes that column's own data streams and is skipped for every part once the column is gone (canSkipMutationCommandForPart). RENAME COLUMN stays blocked, because the recompression is carried over to the new name and executed there. DROP COLUMN is a barrier command, so on a non-replicated table it now deterministically waits for the queued recompression; on a replica that applies the metadata change first, the recompression is skipped — which is exactly the path ea6566d0727a added. Documented in docs/reference/statements/alter/column.mdx. The test is deterministic now (3× green locally).

Unrelated red: BuzzHouse (amd_debug) — Inconsistent AST formatting for RESTORE ... SETTINGS <name> = DEFAULT, tracked as #112895, fix in progress in #112978. Nothing to do here.

Local validation on this head: ninja clickhouse exit 0; all 24 *recompress_column* .sql tests match via clickhouse local --path except 04402_recompress_column, which needs system.part_log (not available in clickhouse local).

🕵 Pushed 1f9cef9129c0 (master merge 975c9c74c47b + the docs move).

AI review Major / Docs check (Mintlify) — fixed. The ALTER TABLE ... COLUMN reference page is generated: its body sits inside the {/*AUTOGENERATED_START*/} region and is rendered from the statement registration in src/Parsers/ParserAlterQuery.cpp. The RECOMPRESS COLUMN documentation was added to the generated .mdx directly, which the No direct edits to generated or read-only docs sub-check rejects and the next autogeneration pass would overwrite. The syntax line, the action list entry, the section and the syntax summary are moved into the registration, and docs/reference/statements/alter/column.mdx is restored to master's content, so this pull request no longer touches the generated region at all. ParserAlterQuery.cpp compiles.

Docs examples — cleared by the master merge. The failure was 1 known failure(s) no longer exist and must be removed: Function/catboostEvaluate#0: #109710 (Remove CatBoost integration, merged 2026-08-26 14:36 UTC) made the tests/docs_examples/known_failures.txt entry stale, and the previous head merged master between that and its revert #116566 (merged 2026-08-26 20:09 UTC). The merge in this push picks up the revert, so the entity exists again and the entry is no longer stale. Nothing in this pull request is involved.

Stress test (arm_asan_ubsan, s3) — unrelated. Logical error: (n >= (static_cast<ssize_t>(pad_left_) ? -1 : 0)) && (n <= static_cast<ssize_t>(this->size())) (STID 2508-35ce) aborts in DB::IEJoinAlgorithm::flushPendingPairs at src/Processors/Transforms/IEJoinTransform.cpp:1038, via IMergingTransform<IEJoinAlgorithm>::work — an IE JOIN path this pull request does not touch. It is the same family as #116805 (Do not pass a chunk with no rows to merging algorithms as data), which fixes IMergingTransform handing a row-less chunk to the algorithm as data; that pull request is open.

@groeneai, could you confirm whether #116805 also covers the IEJoinAlgorithm::flushPendingPairs shape (STID 2508-35ce, report: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=109453&sha=81331f4375036d238bfd332135e39b907316a138&name_0=PR&name_1=Stress%20test%20%28arm_asan_ubsan%2C%20s3%29), and extend it or open a separate fix if it does not?

No, #116805 does not cover it, and no separate fix is needed: #114325 does.

It is not a row-less chunk. IEJoinAlgorithm is constructed with empty_chunk_on_finish = true
(IEJoinTransform.cpp:1246), so its finished signal already arrives as consume with an empty
Input, and consume accumulates a chunk only if (input.chunk.getNumRows() > 0). A row-less
chunk handed over as data therefore adds no rows and leaves num_side_rows and matched
untouched; #116805 only turns it into that same finished signal one step earlier. It does not touch
IEJoinTransform.cpp, and an ie_join arm added to it could not redden.

What overflows is the residual mask. evaluateResidualMask passes its candidate-pair count to
ExpressionActions::executeOnColumns, which takes size_t & num_rows by reference and writes the
post-execution count back; ARRAY_JOIN is the only action that does (ExpressionActions.cpp:742).
The mask is sized from that written-back value while pending_left_data still holds one entry per
pair, so flushPendingPairs walks past its end, reads the padding and indexes matched[0] with the
garbage row id. That is the PODArray<char8_t, ...> frame at IEJoinTransform.cpp:1038.

The fuzzed query carries arrayJoin in the ON clause (carrier 04542_ie_join_float_nan_inf.sql,
bf16 left):

SELECT count() FROM bf_l AS l LEFT JOIN bf_r AS r
    ON and(l.y > r.y, h3EdgeLengthM(arrayJoin([0, 1, 2])), l.x < r.x);

On master 438a6180ae96 that gives the reported assertion with the identical frame, and the
expansion factor is the discriminator: arrayJoin([0]) returns 176, [0, 1] and [0, 1, 2] abort.

#114325 refuses such a condition at the sole writer of ie_join_residual_condition
(JoinStepLogical.cpp:1615), and separately sizes the mask from the pair count and rejects a
mismatch. The abort itself proves the guard fires here: only ARRAY_JOIN can have changed the row
count, so the residual DAG holds one and hasArrayJoin is true. @ vdimir approved it on 2026-08-11;
its one red is an unrelated msan hung check owned by #108357.

🕵 Pushed edc7508e42ae (master merged again + two fixes).

1. Build (arm_tidy) red — this dropped all 145 other jobs, so there is no fresh test signal on the previous head.
src/Storages/MergeTree/MergeTreeData.cpp:93: error: duplicate include [readability-duplicate-include,-warnings-as-errors]

• the previous master merge left a second #include <Parsers/ASTIdentifier.h> in the include block.
  Removed the duplicate; checked every other file this PR touches for the same artifact
  (sort | uniq -d over the #include lines) - none.

2. AI-review Blocker (drop-and-re-add lifecycle of a queued RECOMPRESS COLUMN) - guarded.
canSkipMutationCommandForPart now also skips the recompression while the drop of its target is
still a pending alter conversion for the part, using the same
AlterConversions::isColumnDropped signal the read path uses in
injectRequiredColumnsRecursively. So a same-name ADD COLUMN can no longer re-arm an older
recompression on a part that still carries the pre-drop stream. Details and the reachability
analysis are in the thread reply: #109453 (comment)

Local validation: ninja clickhouse exit 0; all 23 *recompress_column* .sql tests MATCH via
clickhouse local --path (04402_recompress_column is not runnable there - it needs
system.part_log).

Standing rebuttals unchanged (re-point rather than re-implement): KILL MUTATION ON CLUSTER
(r3661111353), DROP-dependent-in-the-same-ALTER (r3698345345), DEFAULT dependents
(r3839741076).

🕵 Status update (2026-09-04):

• The branch was CONFLICTING with master. Merged origin/master (886fb14); the only conflict was a comment block in MutateTask.cpp next to the whole-part-rewrite lossy-codec guard, resolved by keeping the guard and master's shortened comment. All touched translation units and the clickhouse binary rebuild cleanly.
• Addressed the AI review's remaining ask (the KILL MUTATION ON CLUSTER RBAC coverage) in 34d9e8b with a new test 05076_kill_mutation_on_cluster_alter_recompress_column.sh; details and the note on the initiator's all-privileges semantics are in the thread.
• CI reds on the previous head edc7508e42ae:
  • Integration tests (amd_asan_ubsan, db disk, old analyzer, 5/6) and 6/6: both are the job-level Container memory budget exceeded (/docker) - infrastructure/resource failure, no test failure.
  • Stateless tests (amd_binary, flaky check): 04600_recompress_column_nested_shared_offsets hit the 300 s client timeout once on the two-sibling RECOMPRESS COLUMN while waiting with mutations_sync = 2. This test has 1 failure in 6881 CI runs. I audited MergeTreeColumnRecompression.cpp and the MutateTask hookup for that scenario (index_granularity 11, large compress blocks, shared n.size0 stream): the work is strictly linear in bytes and marks, blocks are re-emitted 1:1 and decompressed once, and there is no lock, wait or retry that could stall, so I attribute it to a saturated background pool in the flaky-check run rather than to the feature. Will watch it on the fresh run.

🕵 Merged master (the branch was 3831 commits behind) and fixed the only CI failure.

All 10 red stateless jobs were the pull request's own lossy tests. Master made allow_experimental_codecs obsolete and now gates each experimental codec through its own setting: CompressionCodecFactory looks up getGateSettingName for the codec family and throws Codec SZ3 is experimental and not meant to be used in production. You can enable it with the 'enable_sz3_codec' setting (BAD_ARGUMENTS). All twelve lossy RECOMPRESS COLUMN tests still used the obsolete setting, so every CREATE TABLE / MODIFY COLUMN carrying CODEC(SZ3(...)) was rejected. The two shell tests additionally burned their whole 600 s budget: their table was never created, so the loops polling system.mutations for the expected failure reason never saw it.

Fix: SET enable_sz3_codec = 1 instead of SET allow_experimental_codecs = 1, and --enable_sz3_codec 1 instead of --allow_experimental_codecs 1 for the client invocations — the same way the master tests 03202_sz3_codec and 04791_lossy_codec_key_column do it.

No source change was needed; ninja clickhouse on the merged tree is clean. Locally all 12 lossy tests pass, and 27 of the 30 *recompress_column* tests pass against a server built from this branch (04402_recompress_column needs system.part_log, 04643_recompress_column_pending_rename needs ZooKeeper and 05076_kill_mutation_on_cluster_alter_recompress_column needs a cluster — none of which a minimal local server has; all three are green in CI).

No unresolved review threads remain, and the last AI review verdict was ✅ no blocking findings. Waiting for a fresh CI round on 0c9491a4b0ba.

📊 Cloud Performance Report

✅ AI verdict: no_change — no significant changes across 39 queries analysed

no significant changes detected. K_source=6 K_base=30 flagged=0/65

clickbench

🟢 No significant changes

tpch_adapted_1_official

🟢 No significant changes