Repository navigation
Conversation
_HasIds previously required every column to have an ID. A file with IDs on only some columns, such as a Databricks UniForm table with row-tracking columns, fell back to name mapping and failed when physical names differed from the mapped names. Match Java's ParquetSchemaUtil.hasIds (any) and prune fields without IDs, including incomplete nested types.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4417
Rationale for this change
pyarrow_to_schemaonly used field IDs when every column had one (_HasIdsusedall). If any column lacked an ID, it ignored all IDs and fell back to the name mapping. That fails when physical names differ from the mapped names, which is what happens with Databricks UniForm files that attach IDs to data columns but not to row-tracking columns.I changed
_HasIdsto match Java'sParquetSchemaUtil.hasIds(any), and_ConvertToIcebergnow drops fields without IDs, including lists or maps whose element, key, or value is missing an ID.Are these changes tested?
The issue's example is a unit test in
tests/io/test_pyarrow_visitor.py, with and without a name mapping. I also added a nested case: a map missing child IDs is dropped, and a struct keeps only the children that have IDs. Existing_HasIdstests for partial IDs now expectTrue. Two schema-compatibility tests that rebuild a nested field fromas_arrow()now keep field IDs on that field, so they still test a missing nested column rather than the new prune path.pytest tests/io/test_pyarrow_visitor.pyand the related schema-compatibility tests passed.make lintpassed.Are there any user-facing changes?
Files that already have IDs on every column are unchanged. Files with IDs on only some columns are now read by ID, and columns without IDs are ignored, even if a name mapping is present. That also applies to schema-compatibility checks on write: an extra column without an ID is dropped rather than rejected. I can change that if you'd rather keep the old write-path behavior.
Happy to adjust the approach if you'd prefer something else.