Combine small blocks into one Arrow IPC record batch on output - #119873

ArrowIPCBlockOutputFormat::consume wrote exactly one record batch per incoming
chunk, and nothing squashes on the SELECT ... FORMAT path:
markOutputFormatPrefersLargeBlocks("ArrowStream") is only consulted through
IStorage::prefersLargeBlocks() on the INSERT path. A selective query therefore
emitted one small batch per scanned block, and the fixed FlatBuffers metadata
and buffer padding of each batch dominated the payload. On the reported example,
SELECT number FROM numbers(268435456) WHERE number % 65536 = 0 FORMAT
ArrowStream produced 721,040 bytes in 4,096 one-row batches; at
output_format_arrow_record_batch_size = 65409 and
output_format_arrow_record_batch_size_bytes = 1048576 it produces 16,728 bytes
in one batch, byte-identical to a hand-written single-batch encoding of the
same rows.

consume becomes a staging front end and its former body moves verbatim into
writeChunk, which is also the seam PR ClickHouse#117754 rewrites to split an oversized
chunk, so either merge order is a mechanical rebase and the two compose in the
right order: accumulate up to the target, then split what a single batch still
cannot address.

Sub-target chunks are appended into one accumulator built from
cloneEmptyColumns() rather than from the first chunk: FilterTransform passes
result_size_hint = -1, so ColumnVector::filter reserves the whole source block
and resize_exact does not release it, and adopting the first chunk would make
the accumulator carry that reservation for the lifetime of every batch. A chunk
that already reaches a target flushes what is staged and is then written
un-copied, which keeps a large block out of the accumulator and bounds a
combined batch below twice the target; gating that branch on an empty
accumulator instead would let a few staged rows absorb a 1,048,449-row INSERT
block.

The size is measured on the materialized chunk because ColumnConst::byteSize
reports one stored value, and materializeChunk is a precondition of appending
at all: MessageQueueSink drives this writer through IOutputFormat::write with
no MaterializingTransform in front of it, and appending into an accumulator
cloned from a ColumnConst would advance the row count without copying values.
The targets bound the accumulated data as Chunk::bytes() measures it, on the
same terms as min_insert_block_size_bytes, not retained memory: a
LowCardinality shared dictionary and an aggregate state's foreign arenas sit
outside that measure. Measured against blocks made equally large with
max_block_size instead, the accumulator adds between -1.0 and +1.7 MiB for a
1 MiB target.

The precedent and the refuted alternative, for the record:
output_format_parquet_row_group_size_bytes stages on the cruder
chunk.allocatedBytes() (ParquetBlockOutputFormat.cpp:121) and under-counts
LowCardinality the same way, because byteSize() counts the deduplicated
dictionary once while the encoder writes one value per row unless
output_format_arrow_low_cardinality_as_dictionary is set. Summing each incoming
chunk's bytes as Squashing does was rejected, not overlooked:
ColumnLowCardinality::filter keeps the whole source dictionary and filters only
the indexes (ColumnLowCardinality.h:126-130), so over the 4,096 one-row chunks
this change exists to combine, that sum over-counts by about the chunk count and
would flush at the first one. The byte setting's description states the measure
and advises pairing it with the row criterion.

Both settings default to 0, which is byte-identical to the previous behaviour,
and SettingsChangesHistory records 0 -> 0 so compatibility below 26.9 keeps it.

Closes: ClickHouse#119815

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`ColumnLowCardinality::byteSize` counts one index per row plus each distinct
value once, and nothing at all for the values while the dictionary is still
shared with the block the column came from (ColumnLowCardinality.h:196-197, and
the empty-plus-shared adoption at ColumnLowCardinality.cpp:245-250). That has
two consequences for `output_format_arrow_record_batch_size_bytes` and only one
of them was documented:

- a batch of repeated values encodes to more than the target, because the record
  batch writes one value per row;
- a block filtered out of a larger one keeps that block's dictionary, so it can
  reach the target on its own. Measured on a four-block fixture with one
  surviving row per block: the byte target ends every batch at that single row
  and combines nothing, and since whichever target is reached first ends a
  batch, it also ends the batches a paired row target would have combined.

This is the accounting of `min_insert_block_size_bytes`, which the setting
documents itself against: `Squashing::oneMinReached` uses the same
`Chunk::bytes` measure, the same per-chunk pre-check and the same disjunction of
the two criteria (Squashing.cpp:395-411), and it produces identical block counts
on that fixture in all three configurations. The behaviour is therefore
unchanged; the setting now states what it counts, and 05211 pins both halves
with a `String` control and the row-target remedy beside them.

The `staged` comment no longer claims the accumulator holds none of the source
block's allocations, which the shared-dictionary path would falsify.