Skip to content

Partition field summary bounds ignore the sign of zero #4323

Closed

Description

@karubian

Apache Iceberg version

main (development)

Please describe the bug 🐞

Readers that prune manifests with Iceberg Java can skip rows in tables PyIceberg wrote. If the largest partition value in a manifest is zero, for an identity partition on a double column, PyIceberg can record the upper bound as -0.0 even though the manifest holds 0.0. Java's ManifestEvaluator compares bounds with Double.compareTo, so for x = 0.0 it decides the manifest can't match and skips it: https://github.com/apache/iceberg/blob/b56b4bf9d6f3f4547fd3773e8fd78239869fad4b/api/src/main/java/org/apache/iceberg/expressions/ManifestEvaluator.java#L245-L265

PartitionFieldStats.update takes the summary bounds with Python's min and max, which treat -0.0 and 0.0 as equal and keep whichever came first:

def update(self, value: Any) -> None:
if value is None:
self._contains_null = True
elif isinstance(value, float) and math.isnan(value):
self._contains_nan = True
else:
if self._min is None:
self._min = value
self._max = value
else:
self._max = max(self._max, value)
self._min = min(self._min, value)
. 0.12.0 has the same code.

So for an identity partition on a float or double column holding both zeros, the bounds in the manifest list depend on entry order:

from pyiceberg.conversions import from_bytes
from pyiceberg.manifest import construct_partition_summaries
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.typedef import Record
from pyiceberg.types import DoubleType, NestedField

schema = Schema(NestedField(1, "x", DoubleType(), required=False))
spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=IdentityTransform(), name="x"))
for values in ([0.0, -0.0], [-0.0, 0.0]):
    (summary,) = construct_partition_summaries(spec, schema, [Record(v) for v in values])
    print(values, from_bytes(DoubleType(), summary.lower_bound), from_bytes(DoubleType(), summary.upper_bound))
[0.0, -0.0] 0.0 0.0
[-0.0, 0.0] -0.0 -0.0

The spec rules out both: "If -0.0 is a value of the partition field, the lower_bound must not be +0.0, and if +0.0 is a value of the partition field, the upper_bound must not be -0.0." https://iceberg.apache.org/spec/#manifest-lists

Iceberg Java's PartitionSummary compares with Comparators.forType, which is Double.compareTo for doubles and puts -0.0 below 0.0:

Both orders should give a lower bound of -0.0 and an upper bound of 0.0. #4274 fixes a different signed zero bug, rows written twice on append, and doesn't touch the summaries.

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions