Skip to content

Add files support for parquet field_ids #2131

Closed

Description

@MrDerecho

Feature Request / Improvement

Would it be possible to allow for parquet field_id support in the add_files method? Parquet field id's are a requirement for backwards compatibility with spark based tools- and also, files created by these systems would have field_ids present in the parquet files. This would allow for mass-migrations of spark generated iceberg objects without SerDe (reprocessing the parquet files) to be supported by pyiceberg.

Activity

  1. kevinjqliu commented on Jun 22, 2025

    @kevinjqliu
    Contributor

    allow for parquet field_id support in the add_files method?

    this is already done automatically based on the table metadata.

    if self.table_metadata.name_mapping() is None:
    self.set_properties(
    **{TableProperties.DEFAULT_NAME_MAPPING: self.table_metadata.schema().name_mapping.model_dump_json()}
    )

    You can also specify your own name-mapping by updating the table properties

  2. Erigara commented on Jun 25, 2025

    @Erigara
    Contributor

    If you try to add parquet files which already have field ids you would get this error.

  3. ForeverAngry commented on Jun 28, 2025

    @ForeverAngry
    Contributor

    I think @MrDerecho is wanting a feature added so that add_files() ignores field ids that are present in existing parquet files.

  4. jeroko commented on Oct 27, 2025

    @jeroko
    Contributor

    I was also affected by this constraint, so I created a PR to relax it and only fail if the field IDs from the file are not compatible with the table's field IDs.

  5. added a commit that references this issue on Dec 8, 2025
    454c17f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions