You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
add_column changes the values of an existing column for files without field IDs #4318
0.12.0 (latest release); the same logic is on main
Please describe the bug 🐞
Adding a column changes what an existing column reads. Files added with add_files have no field IDs and are read through the name mapping. After a rename, the renamed column reads correctly. Adding a column that reuses the old name then moves that name to the new field in the mapping (_remove_reassigned_names in pyiceberg/table/name_mapping.py). From then on, the renamed column reads as null for those files, and the new column gets its values.
The manifest statistics of those files still describe the original field, so planning also prunes with statistics that no longer match what is read: below, IsNull("b") skips the file although both rows read b as null.
importtempfileimportpyarrowaspaimportpyarrow.parquetaspqfrompyiceberg.catalog.sqlimportSqlCatalogfrompyiceberg.expressionsimportIsNullfrompyiceberg.schemaimportSchemafrompyiceberg.typesimportLongType, NestedFieldd=tempfile.mkdtemp()
catalog=SqlCatalog("c", uri=f"sqlite:///{d}/c.db", warehouse=f"file://{d}")
catalog.create_namespace("ns")
table=catalog.create_table("ns.t", Schema(NestedField(1, "a", LongType())))
# a Parquet file without field IDs, read through the name mappingpq.write_table(pa.table({"a": [1, 3]}), f"{d}/imported.parquet")
table.add_files([f"file://{d}/imported.parquet"])
withtable.update_schema() asupdate:
update.rename_column("a", "b")
print(table.scan().to_arrow().to_pydict())
# {'b': [1, 3]}withtable.update_schema() asupdate:
update.add_column("a", LongType())
print(table.scan().to_arrow().to_pydict())
# {'b': [None, None], 'a': [1, 3]}: expected {'b': [1, 3], 'a': [None, None]}print(table.scan(row_filter=IsNull("b")).to_arrow().num_rows)
# 0: the file is pruned by the statistics of `b`, which hold its original values
Expected: adding a column leaves the values of existing columns unchanged, as it does for files with field IDs. The name was written for the original field in those files, so either the mapping keeps it there, or adding a column whose name files without field IDs still use is refused.
Willingness to contribute
I can contribute a fix for this bug independently
I would be willing to contribute a fix for this bug with guidance from the Iceberg community
I cannot contribute a fix for this bug at this time
Activity