Skip to content

Report NaN value counts for written Parquet files - #4401

Merged
Fokko merged 4 commits into
apache:mainfrom
Anurag-S1ngh:fix/4396-nan-value-counts
Nov 18, 2026
Merged

Fokko merged 4 commits into
apache:mainfrom
Anurag-S1ngh:fix/4396-nan-value-counts

Conversation

@Anurag-S1ngh

@Anurag-S1ngh Anurag-S1ngh commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Data files written by the PyArrow writer always had an empty nan_value_counts map, because the counts were never computed. The scan planner supports NaN counts, so files written by pyiceberg could never be pruned on is_nan, not_nan, or range filters, while files written by parquet-mr could.

Changes

  • Count the non-null NaNs of the top-level float and double columns from the Arrow table the writer receives, since PyArrow does not expose a NaN count on its Parquet column statistics.
  • Accumulate the counts across batches and pass them into data_file_statistics_from_parquet_metadata, so all the statistics logic stays in one place.
  • Skip columns whose metrics mode is none, like the value and null counts, and compute the statistics plan once per writer.
  • Leave a column out of the map when it cannot be counted, which only means the file cannot be pruned on NaN counts.

Nested float and double columns are not counted yet; that needs flattening through lists, structs, and maps and can follow separately.

The parquet_file_to_data_file (add_files) path still reports empty NaN counts, since PyArrow does not expose a NaN count in the Parquet column statistics. That path needs a separate change, so #4396 stays open.

Testing

  • Added a test asserting the NaN counts of a written file, and one asserting d IS NAN now prunes a file without NaNs.
  • Added tests for counts accumulated across multiple write() calls, for a column with metrics mode none, and for metrics mode none as the default.
  • Updated the merge manifests integration test for the NaN counts on written files.
  • make test passes (4697 passed, 5 skipped).

Part of #4396.

Data files written by the PyArrow writer always had an empty
nan_value_counts map, because the counts were never computed from the
Parquet metadata. The scan planner supports them, so files written by
pyiceberg could never be pruned on is_nan, not_nan, or range filters,
while files written by parquet-mr could.

PyArrow does not expose a NaN count on its Parquet column statistics, so
count the non-null NaNs of the top-level float and double columns in the
Arrow table the writer receives, and merge them into the statistics of the
file. Columns that cannot be counted are left out of the map, which only
keeps them from being pruned.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up! Computing the counts at write time looks like the right workaround while PyArrow doesn't expose nan_count on the column statistics. I left a few comments. The first two need fixing before we merge: one can make pruning skip files that do contain NaNs, and the other ignores the metrics mode.

Also, parquet_file_to_data_file (the add_files path) still returns empty NaN counts, and #4396 mentions that path. Could you either keep the issue open or note in the description that add_files is out of scope?

Comment thread pyiceberg/io/pyarrow.py Resolved
Comment thread pyiceberg/io/pyarrow.py Resolved
Comment thread pyiceberg/io/pyarrow.py Resolved
Comment thread pyiceberg/io/pyarrow.py Resolved
Comment thread pyiceberg/io/pyarrow.py Resolved
Comment thread tests/io/test_pyarrow.py Resolved
@Anurag-S1ngh

Copy link
Copy Markdown
Contributor Author

Thanks Fokko, all addressed.

  • Counts are now added across write() calls instead of replaced, with a test that writes two batches to one file.
  • The statistics plan is computed once in __init__ and passed into the NaN counter, which skips columns whose metrics mode is none. write.metadata.metrics.default=none now gives an empty nan_value_counts. pyiceberg does not implement write.metadata.metrics.max-inferred-column-defaults yet, so there was nothing else to skip.
  • Dropped the try/except and the count is not None check. pc.is_nan and pc.sum do not raise for float or double columns, and pc.sum with min_count=0 always returns a valid scalar.
  • The counts are passed into data_file_statistics_from_parquet_metadata as an optional argument, so the add_files path can reuse it later.
  • Added the tests you asked for and updated the merge manifests integration expectation.

add_files is out of scope here and #4396 stays open; I noted it in the description. Thanks for the nested fields note, I will follow up with structs counted and fields inside lists and maps left out.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Anurag-S1ngh for addressing all the comments, LGTM! This also stops delete() from dropping NaN rows in files written by pyiceberg, since the strict metrics evaluator treated missing NaN counts as no NaNs.

@Fokko
Fokko added this pull request to the merge queue Nov 18, 2026
Merged via the queue into apache:main with commit d5b740d Nov 18, 2026
20 checks passed
rahulsmahadev added a commit to rahulsmahadev/iceberg-python that referenced this pull request Nov 18, 2026
Resolve the adjacent additions in pyiceberg/io/pyarrow.py: keep
_nan_value_counts_from_arrow from apache#4401 and _file_parquet_path_to_id_mapping.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants