Split an oversized chunk across several Arrow record batches - #117754

Arrow addresses its `Utf8`, `Binary` and `List` buffers with 32-bit
offsets, so a chunk holding more than 2 GiB of string data (or more than
2^31 list elements) cannot be written as a single record batch. Both
writers used one batch per chunk and failed on such a chunk:

    SELECT randomString(40000) FROM numbers(60000) FORMAT ArrowStream
    Arrow IPC string offset exceeds 32 bits. (TOO_LARGE_ARRAY_SIZE)

    SELECT count() FROM arrowFlight('127.0.0.1:8890',
        '(SELECT randomString(40000) AS s FROM numbers(60000))')
    Error with a Arrow column "String": Capacity error:
    array cannot contain more than 2147483646 bytes, have 2147520000.

Both now split such a chunk into as many batches as it needs, which is
transparent to readers and keeps the schema on 32-bit offsets, so the
output stays readable by every Arrow implementation.

`maxRowsFittingOneArrowBatch` computes how many rows fit one batch. It
walks the column structure rather than the data: a binary search over
`ColumnString` offsets, an offsets translation plus recursion for
`Array`/`Map`, `min` over `Tuple` elements, and nothing at all for the
fixed-width types. `LowCardinality` and `Variant` need a row scan, but
of integers only. Two top-level columns never share a buffer, so the
smallest per-column limit is the batch's limit.

The `String` encoder no longer assembles the data buffer in a temporary
when the column has no null map: the bytes of the range are contiguous
in `ColumnString::getChars()` and the ClickHouse offsets already are the
Arrow ones, so they go straight into the body. With a null map the
temporary is reserved up front instead of grown by doubling. This cuts
the peak memory of writing a 2 GiB String column from 10.4 GB to 6.3 GB.

Closes: #65723

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The previous wording claimed the result is only ever conservative. That
is wrong when `LowCardinality` is written as an Arrow dictionary: the
emitted dictionary bytes do not depend on the row count, because
`ColumnLowCardinality::insertRangeFrom` keeps a shared source dictionary
whole, so every slice re-emits all of it. A chunk whose dictionary alone
exceeds one buffer is therefore not covered by splitting - as it was not
before splitting existed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`LowCardinality(FixedString(n))` was exempted from the row limit because
only a `String` dictionary can hold a variable-width value. But a writer
materializes the column, and with
`output_format_arrow_fixed_string_as_fixed_byte_array = 0` the resulting
`FixedString` goes through 32-bit offsets, so a multi-row batch over the
cap threw instead of splitting:

    SELECT toFixedString(toString(number % 3), 100000)
               ::LowCardinality(FixedString(100000))
    FROM numbers(21475)
    SETTINGS output_format_arrow_fixed_string_as_fixed_byte_array = 0
    FORMAT ArrowStream
    Cannot write a value of 2147500000 bytes to Arrow IPC ...

`maxRowsForLowCardinality` now receives the mode and applies the plain
`FixedString` cap to the materialized width.

`buildArrowListArrayWithArrayColumnData` narrowed the *absolute*
`ColumnArray` offsets to int32 before rebasing them against
`values_start`. That was equivalent while every range started at row 0,
but a split feeds it a non-zero start, so offsets `[1500000000,
2300000000]` rejected the second batch even though its rebased slice
`[0, 800000000]` fits one `List` buffer. The offsets stay 64-bit and the
limit is now checked on the rebased value, where it applies.

The error message and the comment above it claimed only a single
oversized row could reach the throw, which the `LowCardinality` shape
above disproves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Splitting an oversized chunk cut every column for each batch, because
`RecordBatchEncoder` could only encode a whole column from row 0. For a
2 GiB `String` column that held three copies of the data at once - the
chunk, the cut, and the body - and the stateless test hit the container
memory limit in CI:

    05055_arrow_split_large_record_batch.sh: line 18: Killed
        SELECT number, repeat('x', 100000) FROM numbers(21475)
        FORMAT ArrowStream

The encoder now takes `[begin, end)` throughout, so a batch reads its
rows straight out of the chunk. Peak memory for that test drops from
10.4 GB before any of this work, to 6.3 GB, to 4.4 GB.

Offsets have to be rebased against the row before the range: `String`
against `offs[begin - 1]`, `List`/`Map` against `ch_offsets[begin - 1]`
(and the range translated into the child's element range), and a dense
union's offsets against each alternative's own start, which comes from
the same per-discriminator scan `ColumnVariant::updateHashWithValueRange`
uses. Only `Const`, `ColumnReplicated` and `LowCardinality` still cut,
which is cheap: those cuts resize a constant or slice indexes without
copying values.

`consume` now substitutes dictionaries once per chunk rather than once
per batch, so a chunk's dictionary messages precede all of its record
batches instead of interleaving with them.

Verified by building with `MAX_ARROW_BUFFER_SIZE = 24` to force splits on
small data and round-tripping the whole type matrix, checking the batch
counts with pyarrow.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>