Repository navigation
Keep rows in truncate partitions near the minimum int or long - #4407
Anurag-S1ngh wants to merge 6 commits into
Conversation
The writer, like Java, truncates with two's complement arithmetic, so the values from the minimum of the type wrap to a partition value at the top of the range. The Python transform did not wrap, so scan planning projected a different partition and pruned the file, losing those rows for EqualTo, In, and range filters. Wrap the Python transform into the range of the source type, and add the wrapped partition value to the inclusive projection of ranges that can match it. Greater-than filters need the first partition value after the block that holds the minimum, because the boundary itself wraps. Skip the inclusive projection in the residual evaluator when a transform projects to an expression instead of a single predicate.
rahulsmahadev
left a comment
There was a problem hiding this comment.
Thanks for fixing this. I ran it locally against main (33b4ce5):
- The new and changed tests fail on main and pass here.
- I swept
=,!=,<,<=,>,>=,INandNOT INat both ends of int and long, withtruncate[1000]andtruncate[7], comparingto_arrow()andcount()with the expected rows: 77 wrong results on main, none here. - It also fixes
add_filesfor such a file. On main, adding a Parquet file withx = -2**31to this spec fails withstruct.error: 'i' format requires -2147483648 <= number <= 2147483647; here the file gets partition 2147483296, the value the writer uses. Maybe worth a line in the description.
One problem I think should be fixed here: the strict projection still assumes that truncate partitions sort like the source values, and ResidualEvaluator trusts it, so count() is wrong for files without column bounds (for example with write.metadata.metrics.default=counts):
- New with this change:
transform(MIN + 100)is now 2147483296, so the strict projection ofx < MIN + 100isx_trunc < 2147483296. That holds for every ordinary partition, socount()adds their rows. - Already on main: the strict projection of
x > 0isx_trunc > 0, which holds for the wrapped partition, socount()adds the rows near the minimum.
import tempfile
import pyarrow as pa
from pyiceberg.catalog.sql import SqlCatalog
from pyiceberg.expressions import GreaterThan, LessThan
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import TruncateTransform
from pyiceberg.types import IntegerType, NestedField
tmp = tempfile.mkdtemp()
catalog = SqlCatalog("demo", uri=f"sqlite:///{tmp}/catalog.db", warehouse=f"file://{tmp}/warehouse")
catalog.create_namespace("ns")
schema = Schema(NestedField(1, "x", IntegerType()))
spec = PartitionSpec(PartitionField(source_id=1, field_id=1000, transform=TruncateTransform(1000), name="x_trunc"))
t = catalog.create_table("ns.t", schema=schema, partition_spec=spec, properties={"write.metadata.metrics.default": "counts"})
t.append(pa.table({"x": pa.array([-(2**31), 5], pa.int32())}))
for row_filter in [LessThan("x", -(2**31) + 100), GreaterThan("x", 0)]:
scan = t.scan(row_filter=row_filter)
print(len(scan.to_arrow()), scan.count()) # expected 1 1 for both; this PR prints 1 2 for both, main 0 0 and 1 2to_arrow() is right in both cases because it re-applies the row filter. For < and <= with a boundary in the block of the minimum there is no safe strict projection, so strict_project could return None there; for > and >= it needs to exclude the wrapped value, or also return None. A count() assert on a table with write.metadata.metrics.default=counts would cover both.
Smaller things:
- Java:
TruncateUtil.truncateIntwraps the same way, butProjectionUtil.truncateIntegerdoes not add the wrapped partition, so Java has the same gap for<and>. It would help to say in the description that the range projection goes beyond Java. - #4401, #4402, #4406 and this PR all add tests right after
test_append_to_table_bucketed_by_timeand edit the same import block intests/io/test_pyarrow.py, so each pair conflicts. Placing each test next to related tests would let them merge in any order.
| if boundary + 1 < next_block_start: | ||
| return Or(GreaterThanOrEqual(Reference(name), next_block_start), EqualTo(Reference(name), wrapped_minimum)) | ||
| return _truncate_number(name, pred, transform) | ||
| elif isinstance(pred, BoundGreaterThanOrEqual): |
There was a problem hiding this comment.
No test covers this branch: with if boundary < next_block_start: replaced by if False:, every test still passes, although x >= MIN then skips all ordinary partitions. An assert next to the GreaterThan one in test_int_truncate_projection_near_minimum would cover it:
assert project(GreaterThanOrEqual("id", -(2**63) + 1)) == Or(
GreaterThanOrEqual("id_trunc", -9223372036854775800), EqualTo("id_trunc", wrapped_minimum)
)That test projects the LongType id column, so an IntegerType case would help too; only the scan test covers int.
| inclusive_projection = part.transform.project(part.name, predicate) | ||
| inclusive_result = None | ||
| if inclusive_projection is not None: | ||
| if isinstance(inclusive_projection, UnboundPredicate): |
There was a problem hiding this comment.
TruncateTransform.project() can now return an Or, but Transform.project(), TruncateTransform.project() and _truncate_integer_projection() are still annotated UnboundPredicate | None, and this guard shows that callers relied on that. Widening the annotations to BooleanExpression | None would let mypy point at callers like this one.
#4278 rewrites this block into _eval_projection(projection: UnboundPredicate | None), which calls .bind() on the projection, so whichever of the two PRs merges second needs this guard there too.
Fokko
left a comment
There was a problem hiding this comment.
Thanks @Anurag-S1ngh! Great catch, this is a subtle one: the writer (like Java) wraps the truncate of the type minimum via two's complement, so the partition value sits at the top of the range and scan planning silently pruned those files.
I pulled the branch and verified:
tests/test_transforms.py,tests/expressions/test_projection.py,tests/utils/test_datetime.py,tests/expressions/test_literals.pyall pass- Reproduced the end-to-end scenario:
TruncateTransform(1000)on an int column with valueINT_MIN, scanned withEqualTo,In,LessThan,GreaterThan: correct rows come back - Checked the wrap math against Java's floorMod semantics for int32/int64 at several widths, all consistent
- Confirmed the
isinstance(inclusive_projection, UnboundPredicate)guard in visitors.py is needed and safe: the newOr(...)projections areBooleanExpressions, not predicates, and skipping the inclusive re-check there just falls back to the original predicate (which is correct, only less optimal)
One nit, non-blocking: in _truncate_integer_projection (pyiceberg/transforms.py, around line 1048), the not isinstance(pred.literal, LongLiteral) guard falls back to _truncate_number. Since bound int/long predicates always carry LongLiteral literals in practice, that branch is mostly defensive. If a DecimalLiteral ever landed here, pred.literal.value would be a Decimal and the comparison would still work, so no action needed, just noting the fallback covers it.
CI is fully green. LGTM!
Values from the minimum of the source type truncate to a partition value at the top of the type's range, which does not sort with the other partition values. The strict projection assumed it did, so the residual evaluator could treat every row of a file as matching and count() over-counted files without column bounds. Return no strict projection for less-than predicates that can match the wrapped partition value, and for greater-than predicates when the type wraps at all, falling back to reading and filtering the file.
Keeping it out of the end of the partitioning group avoids conflicts with the other branches that add tests there.
With write.metadata.metrics.default=counts there are no column bounds, so count() relies on the residual. Assert it matches the filtered rows for less-than and greater-than filters at the minimum.
|
Thanks for the detailed check. I reproduced the count() case and fixed it in af385e0.
|
|
Thanks Fokko. I left the LongLiteral guard as is, per your note. The strict projection is now conservative near the wrapped partition (af385e0), since the partition value does not sort with the source values. |
…in-wrap # Conflicts: # pyiceberg/expressions/visitors.py
…in-wrap # Conflicts: # tests/io/test_pyarrow.py
Summary
The writer, like Java, truncates with two's complement arithmetic, so values from the minimum of the type wrap to a partition value at the top of the range. The Python transform did not wrap, so scan planning projected a different partition, pruned the file, and lost those rows for
EqualTo,In, and range filters.Changes
EqualToandIn.count()reads the file instead of trusting its record count.The writer output is unchanged, matching Java and existing tables. This also fixes
add_filesfor a Parquet file whose minimum is the type minimum: on main, adding the file raisedstruct.error: 'i' format requires -2147483648 <= number <= 2147483647; the file now gets the partition value the writer uses.The wrapped-partition range projection goes beyond Java:
TruncateUtil.truncateIntwraps the same way, butProjectionUtil.truncateIntegerdoes not add the wrapped partition, so Java has the same gap for<and>.Testing
truncate[1000]int partition withEqualTo,LessThan, andGreaterThanat the minimum.count()coverage on a table withwrite.metadata.metrics.default=counts, where there are no column bounds.make testpasses (4787 passed, 5 skipped).Closes #4403