Repository navigation
Description
Apache Iceberg version
0.12.0 (latest release)
Please describe the bug 🐞
A truncate partition loses rows whose value is near the minimum int or long. EqualTo, In and range filters such as LessThan(x, 0) all miss them. It reproduces on 0.12.0 and on main (33b4ce5).
import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import EqualTo, LessThan
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import TruncateTransform
from pyiceberg.types import IntegerType, LongType, NestedField
tmp = tempfile.mkdtemp()
catalog = SqlCatalog("demo", uri=f"sqlite:///{tmp}/catalog.db", warehouse=f"file://{tmp}/warehouse")
catalog.create_namespace("ns")
schema = Schema(NestedField(1, "id", LongType(), required=True), NestedField(2, "x", IntegerType()))
spec = PartitionSpec(PartitionField(source_id=2, field_id=1000, transform=TruncateTransform(1000), name="x_trunc"))
t = catalog.create_table("ns.trunc", schema=schema, partition_spec=spec)
rows = [{"id": 0, "x": -(2**31)}, {"id": 1, "x": -(2**31) + 100}, {"id": 2, "x": 5}]
t.append(pa.Table.from_pylist(rows, schema=schema.as_arrow()))
def ids(row_filter):
return sorted(t.scan(row_filter=row_filter).to_arrow()["id"].to_pylist())
print(sorted(task.file.partition[0] for task in t.scan().plan_files())) # [0, 2147483296]
print(ids(EqualTo("x", -(2**31)))) # [] -- expected [0]
print(ids(EqualTo("x", -(2**31) + 100))) # [] -- expected [1]
print(ids(LessThan("x", 0))) # [] -- expected [0, 1]For truncate[1000] on an int, every value from -2147483648 to -2147483001 is affected. A long column behaves the same way near -(2**63).
Cause. The writer computes the partition value with pyiceberg-core. Its math wraps around, as Java's TruncateUtil.truncateInt does. So these rows land in partition 2147483296, a large positive value. Scan planning projects the filter with the Python transform, v - v % self._width, which does not wrap. EqualTo(x, -2147483648) projects to -2147484000, so it never matches the stored value. LessThan(x, 0) projects to x_trunc <= -1000, and the stored value is positive. Either way, the scan prunes the file.
What a fix could look like.
- The writer should keep its current output. Java writes the wrapped value too, so existing tables already hold it.
- Wrapping the Python transform like Java fixes
EqualToandIn. - Range filters need more than that, because the wrapped value does not sort with the others. Java's projection (ProjectionUtil.java#L42-L58) has the same gap. A comment on apache/iceberg#12915 notes the
Integer.MIN_VALUEoverflow, and that issue is closed. API: Integer truncate overflow can prune matching data files iceberg#18111 covers a different overflow in the same function.
I'm not sure of the right fix for range filters. It may need a note in the spec. I can raise it in apache/iceberg if that helps.
How I found it
I used Temper to audit each table layout against the same scan guarantees. The reference is the same filter written as plain Python over the appended rows. Temper helped identify this bug. In MATRIX.md, only the truncate layout fails "int and long extremes are found". The tests are here.
Once the approach is agreed, I can send a PR for EqualTo and In, with regression tests in the existing test files.
I used an AI assistant to run the tests and draft this issue. I verified the example on 0.12.0 and on main, with pyarrow 26.0.1 and Python 3.12.9.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity