Skip to content

Parquet file with field IDs on only some columns falls back to name mapping and fails #4417

Open

Description

@dgokeeffe

Apache Iceberg version

0.12.0 (also fails on main at ce5a195 and on 0.9.1)

Please describe the bug 🐞

PyIceberg can't read a Parquet file in which only some columns have field IDs.

pyarrow_to_schema uses field IDs only when every column has one (_HasIds uses all). If any column lacks an ID, it ignores all IDs and falls back to the name mapping, so the read fails when the physical column names differ from the mapped names.

Iceberg Java treats a file as having IDs if any column has one (ParquetSchemaUtil.hasIds) and prunes the columns without IDs.

Minimal example, no catalog needed:

import pyarrow as pa
from pyiceberg.io.pyarrow import pyarrow_to_schema
from pyiceberg.table.name_mapping import MappedField, NameMapping

file_schema = pa.schema([
    pa.field("col-2b93b371", pa.string(), metadata={"PARQUET:field_id": "1"}),
    pa.field("_row-id-col-922d6355", pa.int64()),  # no field id
])
name_mapping = NameMapping([MappedField(field_id=1, names=["store_id"])])
pyarrow_to_schema(file_schema, name_mapping)
ValueError: Could not find field with name: col-2b93b371

Expected: a schema with field 1, ignoring the column without an ID.

This comes from a Databricks Delta table with UniForm (Iceberg v2), column mapping and row tracking, read through the Unity Catalog Iceberg REST catalog. Its data files have:

  • 13 data columns with physical names like col-<uuid>, each with a field ID
  • 2 row tracking columns, _row-id-col-<uuid> and _row-commit-version-col-<uuid>, without field IDs

tbl.scan().to_arrow() fails with the error above on every file. Spark with the Iceberg 1.12.0 runtime, Polars scan_iceberg and DuckDB read the same table correctly.

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions