Repository navigation
Decode 4-byte bounds of date columns promoted to timestamp - #4409
rahulsmahadev wants to merge 1 commit into
Conversation
Bounds written before a date to timestamp or timestamp_ns promotion keep their 4-byte day count, but from_bytes used the 8-byte decoder for these types, so planning and inspect raised struct.error. Scale the day count to micros or nanos and return the exact int, so metrics comparisons stay exact (metadata tables can still raise OverflowError for bounds outside the Arrow range). Co-authored-by: Isaac <no-reply@databricks.com>
|
@rambleraptor, this extends the 4-byte bound fallback you added in #3293 to date -> timestamp, and @avison9, you added the inspect coverage in #4314. Could you take a look when you have a chance? |
|
Thanks for the ping! I pulled this locally and it looks good to me.
A few non-blocking suggestions from me regarding your fix:
LGTM otherwise. avison9 |
| if len(b) < 8: | ||
| # If the length is 4 bytes, it is a date promoted to timestamp, stored as days from the epoch | ||
| return _INT_STRUCT.unpack(b)[0] * _MICROS_PER_DAY |
There was a problem hiding this comment.
Ideally we don't want to guess based on the number of bytes. But I guess that ship has sailed, since we do the same for the float/double case 😭 @rahulsmahadev Are you intersted in kicking your LLM of to see if we can remove those cases, and actually rely on the type?
There was a problem hiding this comment.
Happy to look into it. One constraint: the spec says readers must infer the write-time type from the bound's length, because manifests do not store the bound type and promotion does not rewrite bounds ("Schema Evolution" in format/spec.md). The manifest's schema metadata is the schema when the manifest was written, which can be newer than the data file when manifests are rewritten, so it cannot replace the length check everywhere. I'll dig into whether there is a reliable source for the write-time type, and open a follow-up PR if there is.
Rationale for this change
Iceberg v3 allows
dateto be promoted totimestampandtimestamp_ns. The promotion does not rewrite thebounds of existing data files, so a bound written before the promotion is still a 4-byte day count from
1970-01-01. The spec says readers must infer the write-time type from the length of the bound: in the
"Schema Evolution" section of
format/spec.md, a 4-bytetimestamportimestamp_nsbound is adate.from_bytesregistersTimestampTypeandTimestampNanoTypewith the 8-byte decoder, so such a bound fails:On a v3 table with retained date bounds after promotion to
timestamp,tbl.scan(row_filter=GreaterThan("d", "2022-01-01T00:00:00")).plan_files()raises thisstruct.errorwhen evaluating those bounds.tbl.inspect.files()raises it after promotion to eithertimestamportimestamp_ns.The fix decodes a 4-byte bound of
TimestampTypeandTimestampNanoTypeas a signed day count, scaled to microseconds (x 86,400,000,000) or nanoseconds (x 86,400,000,000,000), and returns the exact Python int. The 8-byte path is unchanged. The fallback decodes 4-byte bounds; unsupported lengths still raisestruct.error. The change adds no per-file containers or caches.TimeType,TimestamptzTypeandTimestamptzNanoTypekeep their decoder, because the spec does not allowdateto be promoted to them. The fix is infrom_bytes, the one decoder that the inclusive and strict metrics evaluators andinspectall use, so no call site changes.Planning compares such bounds exactly, because Python ints have no int64 limit. Metadata tables are different: converting a bound outside the Arrow range to a timestamp raises
OverflowError. For date bounds promoted totimestamp_ns, representable midnight dates run from 1677-09-22 through 2262-04-11 inclusive; metadata inspection raisesOverflowErrorfor dates outside that interval. The spec's runtime failure rule (Schema Evolution) applies to promoted data values. It does not apply to bounds, because the bounds rules only require a lower bound at or below all values and an upper bound at or above all values, so a bound can be wider than the data (format/spec.md, the data file fieldslower_boundsandupper_bounds). An out-of-range bound therefore does not show that a data value overflows.This PR follows the spec rather than Java. Java's
Conversions.fromByteBuffer(
api/src/main/java/org/apache/iceberg/types/Conversions.java, lines 171-179) returns the raw day count for a4-byte
TIMESTAMPorTIMESTAMP_NANObound without scaling it. Comparing that count with a microsecond literalcould prune a file that has matching rows.
Alternatives considered: B, raise
ValueErrorfor bounds that do not fit in int64. That fails planning or inspection even when every row is representable, because a bound need not occur in the data. C, treat such bounds as unknown, or leave them unavailable in metadata tables. That avoids the errors, but it changes the evaluator and metadata contracts, so it is left for a follow-up. Actual-value overflow checks belong to date-promotion support.This extends #3293, which added the same kind of 4-byte fallback for int to long and float to double bounds, and #4314, which added the inspect test for those types (it closes #3744). Neither covers date to timestamp.
Out of scope: normal schema promotion and reading date data files as timestamps remain unsupported (related closed PR #2586). Comparison filters on
timestamp_nsremain unsupported for string, integer and datetime literals (#4078).Are these changes tested?
Yes.
test_from_bytes_date_bound_promoted_to_timestampintests/test_conversions.pychecks exact micros and nanos for 4-byte day counts, including a date before 1970, and checks that the 8-byte encoding is unchanged.test_from_bytes_date_bound_promoted_to_timestamp_beyond_int64checks the exact scaled value for the first day count outside the int64 range, in both directions.test_plan_and_inspect_after_date_to_timestamp_promotionintests/expressions/test_evaluator.pyuses the public API with each catalog fixture. Two filters plan exactly the expected data file each, andinspect.files()returns the exact bounds. It simulates the promotion withupdate_schema(allow_incompatible_changes=True), because PyIceberg cannot write v3 metadata yet (#1551), and the bound decoder does not depend on the format version. The new rows intest_inclusive_metrics_eval_bounds_after_promotionandtest_strict_metrics_eval_bounds_after_promotioncover both metrics evaluators, including a day count before the epoch. The strict row compares against a literal after the epoch, so a missing scale fails it.Without this change, the new tests fail with
struct.error: unpack requires a buffer of 8 bytes, including the beyond-int64 test. With this change, they pass.Are there any user-facing changes?
Yes. Filtered file planning for
timestampand metadata inspection fortimestampandtimestamp_nsnow decode retained 4-byte date bounds. This does not add date-to-timestamp data reading or nanosecond literal binding. Please add thechangeloglabel.This pull request and its description were written by Isaac.