Repository navigation
Description
Apache Iceberg version
0.12.0 (latest release)
Please describe the bug 🐞
Also reproduced on 0.11.1. main (e9f7f6b) has the same code path.
When PyIceberg writes a Parquet data file, data_file_statistics_from_parquet_metadata (pyiceberg/io/pyarrow.py) builds a column's lower_bounds and upper_bounds from the row-group statistics. If one row group has a statistics object (is_stats_set is true, with a null count) but no min/max for the column, its values are left out of the bounds. The other row groups' bounds are then stored as if they covered the whole file.
PyArrow (25.0.1 here) writes a row group's statistics without min/max when one of its values for that column is larger than 4,096 bytes. A string column with a single long value is enough. There is no writer option in pyarrow.parquet to change that.
The stored bounds are then too narrow. _InclusiveMetricsEvaluator trusts them and drops the file from any scan whose filter value is outside them, so table.scan(row_filter=...) returns no rows even though the rows are stored. The file is not marked as missing statistics: only is_stats_set == False and ArrowNotImplementedError add a column to invalidate_col today.
Reproduction
Self-contained, no services:
import tempfile
import pyarrow as pa
import pyarrow.parquet as pq
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import EqualTo
warehouse = tempfile.mkdtemp()
catalog = SqlCatalog("repro", uri=f"sqlite:///{warehouse}/c.db", warehouse=f"file://{warehouse}")
catalog.create_namespace("ns")
schema = pa.schema([pa.field("k", pa.string())])
table = catalog.create_table("ns.t", schema=schema, properties={"write.parquet.row-group-limit": "10000"})
long_value = "z" * 5000
values = [f"a{i:05d}" for i in range(10000)] + [f"b{i:05d}" for i in range(9999)] + [long_value]
table.append(pa.table({"k": values}, schema=schema))
[task] = table.scan().plan_files()
metadata = pq.ParquetFile(task.file.file_path.removeprefix("file://")).metadata
print([metadata.row_group(i).column(0).statistics.has_min_max for i in range(metadata.num_row_groups)])
print(task.file.lower_bounds.get(1), task.file.upper_bounds.get(1))
print(table.scan().to_arrow().num_rows)
print(table.scan(row_filter=EqualTo("k", "b00005")).to_arrow().num_rows)
print(table.scan(row_filter=EqualTo("k", long_value)).to_arrow().num_rows)Output with 0.12.0:
[True, False]
b'a00000' b'a09999'
20000
0
0
Expected: the last two scans return 1 row each. Either the bounds cover b00000 to the long value (truncated per the metrics mode), or the column has no bounds for that file.
Suggested fix
In data_file_statistics_from_parquet_metadata, treat a row group that holds non-null values for a column but whose statistics have no min/max like a row group without statistics: add the field to invalidate_col. A row group whose values are all null (statistics.null_count == column.num_values) has no min/max either and must keep the current behaviour.
if stats_col.mode == MetricsMode(MetricModeTypes.COUNTS):
continue
if not statistics.has_min_max:
if statistics.has_null_count and statistics.null_count == column.num_values:
continue
invalidate_col.add(field_id)
continueWith this change applied to a copy of 0.12.0, the reproduction prints [True, False], None None, 20000, 1, 1. A file whose first row group holds only nulls ([None, None], then ["b", "c"], with write.parquet.row-group-limit 2) still gets the bounds b'b' and b'c'.
I can open a pull request with this change and a test built from the reproduction above.
Willingness to contribute
- I can contribute a fix for this bug independently
Activity