Repository navigation
Description
Apache Iceberg version
main (development)
Please describe the bug 🐞
Readers that prune manifests with Iceberg Java can skip rows in tables PyIceberg wrote. If the largest partition value in a manifest is zero, for an identity partition on a double column, PyIceberg can record the upper bound as -0.0 even though the manifest holds 0.0. Java's ManifestEvaluator compares bounds with Double.compareTo, so for x = 0.0 it decides the manifest can't match and skips it: https://github.com/apache/iceberg/blob/b56b4bf9d6f3f4547fd3773e8fd78239869fad4b/api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java#L245-L265
PartitionFieldStats.update takes the summary bounds with Python's min and max, which treat -0.0 and 0.0 as equal and keep whichever came first:
iceberg-python/pyiceberg/manifest.py
Lines 727 to 738 in c8d6723
| def update(self, value: Any) -> None: | |
| if value is None: | |
| self._contains_null = True | |
| elif isinstance(value, float) and math.isnan(value): | |
| self._contains_nan = True | |
| else: | |
| if self._min is None: | |
| self._min = value | |
| self._max = value | |
| else: | |
| self._max = max(self._max, value) | |
| self._min = min(self._min, value) |
So for an identity partition on a float or double column holding both zeros, the bounds in the manifest list depend on entry order:
from pyiceberg.conversions import from_bytes
from pyiceberg.manifest import construct_partition_summaries
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.typedef import Record
from pyiceberg.types import DoubleType, NestedField
schema = Schema(NestedField(1, "x", DoubleType(), required=False))
spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=IdentityTransform(), name="x"))
for values in ([0.0, -0.0], [-0.0, 0.0]):
(summary,) = construct_partition_summaries(spec, schema, [Record(v) for v in values])
print(values, from_bytes(DoubleType(), summary.lower_bound), from_bytes(DoubleType(), summary.upper_bound))[0.0, -0.0] 0.0 0.0
[-0.0, 0.0] -0.0 -0.0
The spec rules out both: "If -0.0 is a value of the partition field, the lower_bound must not be +0.0, and if +0.0 is a value of the partition field, the upper_bound must not be -0.0." https://iceberg.apache.org/spec/#manifest-lists
Iceberg Java's PartitionSummary compares with Comparators.forType, which is Double.compareTo for doubles and puts -0.0 below 0.0:
- https://github.com/apache/iceberg/blob/b56b4bf9d6f3f4547fd3773e8fd78239869fad4b/core/src/main/java/org/apache/iceberg/PartitionSummary.java#L84-L99
- https://github.com/apache/iceberg/blob/b56b4bf9d6f3f4547fd3773e8fd78239869fad4b/api/src/main/java/org/apache/iceberg/types/Comparators.java#L40-L41
Both orders should give a lower bound of -0.0 and an upper bound of 0.0. #4274 fixes a different signed zero bug, rows written twice on append, and doesn't touch the summaries.
Willingness to contribute
- I can contribute a fix for this bug independently
- I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- I cannot contribute a fix for this bug at this time
Activity