Skip to content

Populate nan_value_counts when writing parquet files - #4412

Open
AP24110011695 wants to merge 1 commit into
apache:mainfrom
AP24110011695:fix/nan-value-counts-writer
Open

AP24110011695 wants to merge 1 commit into
apache:mainfrom
AP24110011695:fix/nan-value-counts-writer

Conversation

@AP24110011695

Copy link
Copy Markdown
Contributor

What's changed

Implements design A from #4396: data_file_statistics_from_parquet_metadata returned an empty nan_value_counts dict unconditionally, so data files written by the pyarrow writer never reported NaN counts — and NaN-aware scan pruning stayed silently unavailable for them (_InclusiveMetricsEvaluator degrades to ROWS_MIGHT_MATCH without nan counts, for is_nan, not_nan and range predicates via _contains_nans_only).

pyarrow's statistics API does not expose NaN counts (checked on 26.0.1 — no nan_count/has_nan_count), so the writer now counts them from the written table itself:

  • ParquetFormatWriter.write() accumulates pa.compute.is_nan(...) sums per top-level float/double field (is_nan is null on nulls and sum skips nulls — matching the spec's NaN-count semantics of counting non-null NaN values)
  • the counts flow through a new optional table_nan_value_counts parameter of data_file_statistics_from_parquet_metadata
  • the metadata-only caller (parquet_file_to_data_file, used by add_files) keeps its previous behavior — no NaN counts from metadata-only input

Nested float fields keep the previous (empty) behavior for now; the metrics plan and evaluator keys make them a natural follow-up once the top-level path is settled.

Test

test_nan_value_counts_are_written (mixed values + null) and test_nan_value_counts_are_written_across_row_groups (all-NaN, two write() calls) — both verified failing-first on main (nan_value_counts == {}) and passing with the change; the second also pins the pruning win (not_nan now prunes an all-NaN file via _contains_nans_only). Full tests/io/test_pyarrow_stats.py (22) green; the pre-existing azure-fixture failures in test_pyarrow.py are unrelated (missing optional dep in this environment).

AI disclosure: drafted with AI assistance, verified locally by running the failing-first cycle.

data_file_statistics_from_parquet_metadata returned an empty
nan_value_counts dict unconditionally, so data files written by the
pyarrow writer never reported NaN counts and NaN-aware scan pruning
stayed unavailable for them — the metrics evaluator degrades to
ROWS_MIGHT_MATCH without nan counts (_InclusiveMetricsEvaluator
visit_is_nan / _contains_nans_only).

pyarrow's statistics API does not expose NaN counts, so the writer now
counts them from the written table itself (pa.compute.is_nan summed per
top-level float/double field, excluding nulls per the spec's NaN-count
semantics) and passes them through a new optional
table_nan_value_counts parameter. The metadata-only caller
(parquet_file_to_data_file) keeps its previous behavior.

Refs apache#4396.

Signed-off-by: Ayush Kumar Saha <225407166+AP24110011695@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant