You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Parquet file with field IDs on only some columns falls back to name mapping and fails #4417
0.12.0 (also fails on main at ce5a195 and on 0.9.1)
Please describe the bug 🐞
PyIceberg can't read a Parquet file in which only some columns have field IDs.
pyarrow_to_schema uses field IDs only when every column has one (_HasIds uses all). If any column lacks an ID, it ignores all IDs and falls back to the name mapping, so the read fails when the physical column names differ from the mapped names.
Iceberg Java treats a file as having IDs if any column has one (ParquetSchemaUtil.hasIds) and prunes the columns without IDs.
Minimal example, no catalog needed:
importpyarrowaspafrompyiceberg.io.pyarrowimportpyarrow_to_schemafrompyiceberg.table.name_mappingimportMappedField, NameMappingfile_schema=pa.schema([
pa.field("col-2b93b371", pa.string(), metadata={"PARQUET:field_id": "1"}),
pa.field("_row-id-col-922d6355", pa.int64()), # no field id
])
name_mapping=NameMapping([MappedField(field_id=1, names=["store_id"])])
pyarrow_to_schema(file_schema, name_mapping)
ValueError: Could not find field with name: col-2b93b371
Expected: a schema with field 1, ignoring the column without an ID.
This comes from a Databricks Delta table with UniForm (Iceberg v2), column mapping and row tracking, read through the Unity Catalog Iceberg REST catalog. Its data files have:
13 data columns with physical names like col-<uuid>, each with a field ID
2 row tracking columns, _row-id-col-<uuid> and _row-commit-version-col-<uuid>, without field IDs
tbl.scan().to_arrow() fails with the error above on every file. Spark with the Iceberg 1.12.0 runtime, Polars scan_iceberg and DuckDB read the same table correctly.
Willingness to contribute
I can contribute a fix for this bug independently
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
I cannot contribute a fix for this bug at this time
Activity