Account for transformer-retained batches in symmetric hash join - #24364
Draft
kosiew wants to merge 9 commits into
Draft
Account for transformer-retained batches in symmetric hash join#24364kosiew wants to merge 9 commits into
kosiew wants to merge 9 commits into
Conversation
…d batches, deduplicate shared buffers, and update reservation lifecycle** - RecordBatchMemoryCounter now tracks transformer retained batches. - Symmetric stream deduplicates input/transformer shared buffers. - Reservation updates occur on retain and release operations. - Added tests for noop, splitter, shared buffers, and reservation lifecycle.
…te helper (no public API/behavior change)
* Preserve RecordBatch array-object/non-buffer overhead while deduplicating Arrow buffer capacities in memory reservations. * Ensure transformer-held batches are fully accounted for in bounded-memory enforcement. * Add wide/nested retained-batch regression coverage for non-buffer allocation overhead. * Add execution-level bounded-memory tests using SymmetricHashJoinExec with a tight GreedyMemoryPool. * Verify NoopBatchTransformer and BatchSplitter retained batches correctly affect reservations and memory-limit behavior.
…age for children (Struct, List, Map, Union, Dict, RunEnd), plus regression test for shared nested StructArray child
…pBatchTransformer and BatchSplitter
…p contract; add regression-boundary rationale comment for 2_400 test limit
…n, Dictionary, RunEnd and improve shared allocation delta checks
…ArrayRef clones - Introduces a private assertion helper for testing. - Removes unnecessary ArrayRef clones after construction, improving efficiency.
github-actions
Bot
added
common
…n and memory usage - Polls transformer output to exhaustion. - Asserts reservation + stream_memory_usage grow on retain, return baseline on release. - Covers Noop + BatchSplitter.
kosiew
force-pushed
the
memcalc-03-23393
branch
from
August 14, 2026 12:26
ec4a627 to
792fa41
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24364 +/- ##
==========================================
+ Coverage 81.17% 81.19% +0.01%
==========================================
Files 1109 1109
Lines 388117 388591 +474
Branches 388117 388591 +474
==========================================
+ Hits 315071 315528 +457
- Misses 54504 54515 +11
- Partials 18542 18548 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
sizefunctions #23393Rationale for this change
The symmetric hash join can retain an output
RecordBatchin its batch transformer without including that batch in the stream's memory reservation.As a result, the stream can under-account memory while either
NoopBatchTransformerorBatchSplitterholds a batch. The retained batch may also share Arrow buffers or nested arrays with other stream state, so simply adding individual batch memory sizes can double-count shared allocations.This PR accounts for transformer-retained batches at the symmetric hash join stream reservation boundary while deduplicating shared Arrow allocations. Corrected accounting can increase reservation pressure under bounded memory pools, without changing join results.
What changes are included in this PR?
RecordBatchMemoryCounterwith sequence-aware accounting for both Arrow buffers and array-object overhead, including shared nested arrays.BatchTransformer::count_memorysoNoopBatchTransformerandBatchSplitterreport memory retained by their current batch through the shared counter.RecordBatchMemoryCounterwhen calculatingSymmetricHashJoinStream::size(), avoiding duplicate accounting of shared allocations.SymmetricHashJoinStream::update_reservation()and update the reservation when a transformer acquires or releases a batch.OneSideHashJoineraccounting formulas unchanged.Are these changes tested?
Yes. The patch adds the following regression tests:
test_record_batch_memory_counter_array_overhead_shared_across_batchestest_record_batch_memory_counter_deduplicates_shared_nested_array_overheadtest_record_batch_memory_counter_deduplicates_recursive_shared_childrenbatch_transformers_count_retained_batch_memorystream_accounts_for_transformer_batches_oncestream_deduplicates_nested_transformer_batchessymmetric_hash_join_reserves_transformer_batchThese cover transformer-retained batches for both
NoopBatchTransformerandBatchSplitter, shared buffers and nested arrays, reservation/metric updates when transformer ownership changes, and bounded-memory behavior when the previously unaccounted retained batch pushes the join over the configured memory limit.Are there any user-facing changes?
There are no changes to join result semantics or public APIs.
Memory accounting for symmetric hash joins is more complete. Workloads running near a configured memory limit may therefore encounter memory-pressure behavior earlier because transformer-retained batches are now included in the stream reservation.
LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.