Skip to content

Use field IDs when any Parquet column has one and prune the rest - #4419

Open
linzch3 wants to merge 1 commit into
apache:mainfrom
linzch3:fix/4417-partial-parquet-field-ids
Open

linzch3 wants to merge 1 commit into
apache:mainfrom
linzch3:fix/4417-partial-parquet-field-ids

Conversation

@linzch3

@linzch3 linzch3 commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Closes #4417

Rationale for this change

pyarrow_to_schema only used field IDs when every column had one (_HasIds used all). If any column lacked an ID, it ignored all IDs and fell back to the name mapping. That fails when physical names differ from the mapped names, which is what happens with Databricks UniForm files that attach IDs to data columns but not to row-tracking columns.

I changed _HasIds to match Java's ParquetSchemaUtil.hasIds (any), and _ConvertToIceberg now drops fields without IDs, including lists or maps whose element, key, or value is missing an ID.

Are these changes tested?

The issue's example is a unit test in tests/io/test_pyarrow_visitor.py, with and without a name mapping. I also added a nested case: a map missing child IDs is dropped, and a struct keeps only the children that have IDs. Existing _HasIds tests for partial IDs now expect True. Two schema-compatibility tests that rebuild a nested field from as_arrow() now keep field IDs on that field, so they still test a missing nested column rather than the new prune path.

pytest tests/io/test_pyarrow_visitor.py and the related schema-compatibility tests passed. make lint passed.

Are there any user-facing changes?

Files that already have IDs on every column are unchanged. Files with IDs on only some columns are now read by ID, and columns without IDs are ignored, even if a name mapping is present. That also applies to schema-compatibility checks on write: an extra column without an ID is dropped rather than rejected. I can change that if you'd rather keep the old write-path behavior.

Happy to adjust the approach if you'd prefer something else.

_HasIds previously required every column to have an ID. A file with IDs
on only some columns, such as a Databricks UniForm table with
row-tracking columns, fell back to name mapping and failed when
physical names differed from the mapped names.

Match Java's ParquetSchemaUtil.hasIds (any) and prune fields without
IDs, including incomplete nested types.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parquet file with field IDs on only some columns falls back to name mapping and fails

1 participant