You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Would it be possible to allow for parquet field_id support in the add_files method? Parquet field id's are a requirement for backwards compatibility with spark based tools- and also, files created by these systems would have field_ids present in the parquet files. This would allow for mass-migrations of spark generated iceberg objects without SerDe (reprocessing the parquet files) to be supported by pyiceberg.
I was also affected by this constraint, so I created a PR to relax it and only fail if the field IDs from the file are not compatible with the table's field IDs.
Activity
kevinjqliu commentedon Jun 22, 2025
this is already done automatically based on the table metadata.
iceberg-python/pyiceberg/table/__init__.py
Lines 855 to 858 in 89e71c3
You can also specify your own name-mapping by updating the table properties
Erigara commentedon Jun 25, 2025
If you try to add parquet files which already have field ids you would get this error.
ForeverAngry commentedon Jun 28, 2025
I think @MrDerecho is wanting a feature added so that
add_files()ignores field ids that are present in existing parquet files.jeroko commentedon Oct 27, 2025
I was also affected by this constraint, so I created a PR to relax it and only fail if the field IDs from the file are not compatible with the table's field IDs.
Remove `field-id` constraint on `add_files` (apache#2662)