Backport #118796 to 26.7: Iceberg manifest list partition summaries - #119200
Merged
Conversation
Contributor
|
Workflow [BackportPR], commit [6dfa588] Summary: ✅ |
`test_writes_manifest_list_partition_summary` asserts the raw bytes of the manifest-list partition summary, and it is the first test to do so, which is why it fails on this branch for every parametrisation while it is green on master: At index 1 diff: (False, None, b'\x0b\x00\x00\x00\x00\x00\x00\x00', ...) != (False, None, b'\x0b\x00\x00\x00', ...) The `Int32` partition bound is written as 8 bytes where Appendix D of the Iceberg specification requires 4. `dumpValue` sizes its output as `sizeof(T)` with `T` deduced from its argument, and `Field::safeGet<Int32>` returns `NearestFieldType<Int32>`, which is `Int64`, so the branch asks for an `Int32` and gets an 8-byte value. Narrow before `dumpValue` sees it, with the same `static_cast<Int32>` the neighbouring small-integer branch already uses. This is the source half of `3dc6b4f4ef32` (#117571, `Closes #117072`), which never got a backport label; this branch needs it because the test it adds pins the spec width. That test is the coverage here, so the stateless test of the original commit is left out rather than duplicated. The data-file bounds are corrected by the same line, so this branch stops writing manifests that `pyiceberg` refuses to plan a filtered scan over (`unpack requires a buffer of 4 bytes`). No reader change is needed in either direction: `ColumnVector::insertData` ignores the buffer length and does `unalignedLoad<T>(pos)`, so an existing 8-byte bound still decodes. Related: #117571 Related: #117072
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Original pull-request #118796
This pull-request is a last step of an automated backporting.
Treat it as a standard pull-request: look at the checks and resolve conflicts.
Merge it only if you intend to backport changes to the target branch, otherwise just close it.
The PR source
The PR is created in the CI job
Version info
26.7.8.8