Repository navigation
Report NaN value counts for written Parquet files - #4401
Conversation
Data files written by the PyArrow writer always had an empty nan_value_counts map, because the counts were never computed from the Parquet metadata. The scan planner supports them, so files written by pyiceberg could never be pruned on is_nan, not_nan, or range filters, while files written by parquet-mr could. PyArrow does not expose a NaN count on its Parquet column statistics, so count the non-null NaNs of the top-level float and double columns in the Arrow table the writer receives, and merge them into the statistics of the file. Columns that cannot be counted are left out of the map, which only keeps them from being pruned.
Fokko
left a comment
There was a problem hiding this comment.
Thanks for picking this up! Computing the counts at write time looks like the right workaround while PyArrow doesn't expose nan_count on the column statistics. I left a few comments. The first two need fixing before we merge: one can make pruning skip files that do contain NaNs, and the other ignores the metrics mode.
Also, parquet_file_to_data_file (the add_files path) still returns empty NaN counts, and #4396 mentions that path. Could you either keep the issue open or note in the description that add_files is out of scope?
|
Thanks Fokko, all addressed.
|
Fokko
left a comment
There was a problem hiding this comment.
Thanks @Anurag-S1ngh for addressing all the comments, LGTM! This also stops delete() from dropping NaN rows in files written by pyiceberg, since the strict metrics evaluator treated missing NaN counts as no NaNs.
Resolve the adjacent additions in pyiceberg/io/pyarrow.py: keep _nan_value_counts_from_arrow from apache#4401 and _file_parquet_path_to_id_mapping. Co-authored-by: Isaac <no-reply@databricks.com>
Summary
Data files written by the PyArrow writer always had an empty
nan_value_countsmap, because the counts were never computed. The scan planner supports NaN counts, so files written by pyiceberg could never be pruned onis_nan,not_nan, or range filters, while files written by parquet-mr could.Changes
data_file_statistics_from_parquet_metadata, so all the statistics logic stays in one place.none, like the value and null counts, and compute the statistics plan once per writer.Nested float and double columns are not counted yet; that needs flattening through lists, structs, and maps and can follow separately.
The
parquet_file_to_data_file(add_files) path still reports empty NaN counts, since PyArrow does not expose a NaN count in the Parquet column statistics. That path needs a separate change, so #4396 stays open.Testing
d IS NANnow prunes a file without NaNs.write()calls, for a column with metrics modenone, and for metrics modenoneas the default.make testpasses (4697 passed, 5 skipped).Part of #4396.