Repository navigation
Populate nan_value_counts when writing parquet files - #4412
Open
AP24110011695 wants to merge 1 commit into
Open
AP24110011695 wants to merge 1 commit into
AP24110011695 wants to merge 1 commit into
Conversation
data_file_statistics_from_parquet_metadata returned an empty nan_value_counts dict unconditionally, so data files written by the pyarrow writer never reported NaN counts and NaN-aware scan pruning stayed unavailable for them — the metrics evaluator degrades to ROWS_MIGHT_MATCH without nan counts (_InclusiveMetricsEvaluator visit_is_nan / _contains_nans_only). pyarrow's statistics API does not expose NaN counts, so the writer now counts them from the written table itself (pa.compute.is_nan summed per top-level float/double field, excluding nulls per the spec's NaN-count semantics) and passes them through a new optional table_nan_value_counts parameter. The metadata-only caller (parquet_file_to_data_file) keeps its previous behavior. Refs apache#4396. Signed-off-by: Ayush Kumar Saha <225407166+AP24110011695@users.noreply.github.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's changed
Implements design A from #4396:
data_file_statistics_from_parquet_metadatareturned an emptynan_value_countsdict unconditionally, so data files written by the pyarrow writer never reported NaN counts — and NaN-aware scan pruning stayed silently unavailable for them (_InclusiveMetricsEvaluatordegrades toROWS_MIGHT_MATCHwithout nan counts, foris_nan,not_nanand range predicates via_contains_nans_only).pyarrow's statistics API does not expose NaN counts (checked on 26.0.1 — no
nan_count/has_nan_count), so the writer now counts them from the written table itself:ParquetFormatWriter.write()accumulatespa.compute.is_nan(...)sums per top-level float/double field (is_nanis null on nulls andsumskips nulls — matching the spec's NaN-count semantics of counting non-null NaN values)table_nan_value_countsparameter ofdata_file_statistics_from_parquet_metadataparquet_file_to_data_file, used byadd_files) keeps its previous behavior — no NaN counts from metadata-only inputNested float fields keep the previous (empty) behavior for now; the metrics plan and evaluator keys make them a natural follow-up once the top-level path is settled.
Test
test_nan_value_counts_are_written(mixed values + null) andtest_nan_value_counts_are_written_across_row_groups(all-NaN, twowrite()calls) — both verified failing-first on main (nan_value_counts == {}) and passing with the change; the second also pins the pruning win (not_nannow prunes an all-NaN file via_contains_nans_only). Fulltests/io/test_pyarrow_stats.py(22) green; the pre-existingazure-fixture failures intest_pyarrow.pyare unrelated (missing optional dep in this environment).AI disclosure: drafted with AI assistance, verified locally by running the failing-first cycle.