You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
parquet_path_to_id_mapping generates incorrect path for List types #716
When using the add_files table api, the parquet metadata needs to be read and a mapping of Dict[str, int] is used by data_file_statistics_from_parquet_metadata in order to link the field ID to the name in the parquet file for statistics collection. However during the mapping lookup I was receiving an error that a key was not present.
My schema contains one of the following (its a subfield of a Details struct which is important for the full name later):
extras: large_list<item: struct<key: string not null, value: string>> not null
child 0, item: struct<key: string not null, value: string>
child 0, key: string not null
child 1, value: string
Which based on the parquet schema path definition has a path of:
So, the issue appears to be that the visitor for constructing the schema paths is incorrectly using element instead of item as expected in the parquet schema paths. I am not sure how this manifests yet, as I have not dug into it too closely.
Here is a complete example recreating the error. Here I am using polars to make the table which results in the same schema that I am producing with pyarrow.
column mappings 1
{'a.list.element.a': 3}
Traceback (most recent call last):
File "/home/cgbur/pyice-test/failure.py", line 36, in <module>
table.add_files(["example.parquet"])
File "/local/home/cgbur/pyice-test/iceberg-python/pyiceberg/table/__init__.py", line 1355, in add_files
tx.add_files(file_paths=file_paths, snapshot_properties=snapshot_properties)
File "/local/home/cgbur/pyice-test/iceberg-python/pyiceberg/table/__init__.py", line 462, in add_files
for data_file in data_files:
File "/local/home/cgbur/pyice-test/iceberg-python/pyiceberg/table/__init__.py", line 2737, in _parquet_files_to_data_files
yield from parquet_files_to_data_files(io=io, table_metadata=table_metadata, file_paths=iter(file_paths))
File "/local/home/cgbur/pyice-test/iceberg-python/pyiceberg/io/pyarrow.py", line 1869, in parquet_files_to_data_files
statistics = data_file_statistics_from_parquet_metadata(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/local/home/cgbur/pyice-test/iceberg-python/pyiceberg/io/pyarrow.py", line 1734, in data_file_statistics_from_parquet_metadata
field_id = parquet_column_mapping[column.path_in_schema]
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'a.list.item.a'
You can see how the parquet path_in_schema using the item instead of element.
Perhaps the visitor is not respecting the name used in the schema? Or there is a mismatch in the method used to acquire between the iceberg and parquet change? When use_pyarrow=True is set, the example I pasted above does not error out which makes me think something is lost in translation.
Hi, I investigated this a bit further and it seems to be related to the way the visitor works as @cgbur suggested. Here is what I tried:
deftest_parquet_path_to_id_mapping():
# set field name to "item"pyarrow_list=pa.schema([
pa.field("extras", pa.list_(pa.field("item", pa.struct([pa.field("key", pa.string()), pa.field("value", pa.string())]))))
])
# this is called during table creationschema=Catalog._convert_schema_if_needed(pyarrow_list)
mapping=parquet_path_to_id_mapping(schema)
assert"extras.list.item.key"inmapping# raises error
The mapping that Catalog._convert_schema_if_needed creates overrides the field name and sets it to "element":
Hi @cgbur and @felixscherz thank you for raising this and taking this investigation further. I'm not a polars user myself, but the difference in the behavior is quite interesting, and I think there would be value in trying to fix this issue.
I just read the write_table API documentation in pyarrow.parquet and found something rather interesting:
If you check the documentation for use_compliant_nested_type flag, it mentions that having element as the single item field name is the parquet compliant format as specified here on the Parquet Spec for Nested Types. PyArrow defaults to using this flag and writes the list element field name as element.
For some reason it looks like polars has decided to use item - the non-parquet compliant list element name instead. While I'm curious to know why the polars community has decided to go this route and whether this is common practice in the Parquet community, here are some opinions I've formed on this issue for now:
This is an issue we are running into only when we are trying to add the files of a polars produced parquet file as Iceberg data files. We do not have the same issues when writing parquet files using existing Iceberg clients.
While adding support for both element or item name in the function will be very simple, and will allow us to increase our scope of support for add_files API, item is not the correct name for the single value field in the LIST field type according to the Parquet Spec. As a result, adding this as a data file to an Iceberg table may have unintended consequences for other parts of our infrastructure that make similar assumptions in accordance to the Parquet Spec. Hence, my hunch is that we should review the potential downstream impact of this change, carefully.
I believe this is because they are not converting to parquet, but to arrow, then parquet. A subtle difference in their internal logic. However, perhaps the confusion arises because in arrow, the List single element name is often item not element.
I'll go ahead and open an issue on the Polars repo and see if they have anything to say, there are multiple ways to fix this in their package and agree they should likely be producing parquet files according to the spec.
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
Activity
Fokko commentedon May 8, 2024
@cgbur Thanks for raising this. Could you share the stack trace that you're seeing?
I tried to reproduce it, but it works on my end:
cgbur commentedon May 8, 2024
Here is a complete example recreating the error. Here I am using polars to make the table which results in the same schema that I am producing with pyarrow.
And here is the error. The top two lines were debug statements showing how the mapping file has the incorrect path.
You can see how the parquet
path_in_schemausing theiteminstead ofelement.cgbur commentedon May 8, 2024
Ah, confusingly there appears to be writer differences that cause the issue. My Rust pyarrow implementation matches when polars has
pyarrow=True.Perhaps the visitor is not respecting the name used in the schema? Or there is a mismatch in the method used to acquire between the iceberg and parquet change? When
use_pyarrow=Trueis set, the example I pasted above does not error out which makes me think something is lost in translation.felixscherz commentedon May 18, 2024
Hi, I investigated this a bit further and it seems to be related to the way the visitor works as @cgbur suggested. Here is what I tried:
The mapping that
Catalog._convert_schema_if_neededcreates overrides the field name and sets it to "element":{'extras.list.element.key': -1, 'extras.list.element.value': -1}Looking into the visitor I found the method dealing with list types sets a default field name of "elements".
iceberg-python/pyiceberg/io/pyarrow.py
Lines 865 to 870 in 20c2731
iceberg-python/pyiceberg/io/pyarrow.py
Line 172 in 20c2731
So we lose the information on the field name of the value field, setting it to "elements".
Unfortunately I haven't found a way to access the field name as both pyarrow.lib.ListType and pyarrow.lib.DataType don't seem to make that available.We can access the name with the
nameattribute, maybe we can use that instead of going with a default of "element"?sungwy commentedon Jun 15, 2024
Hi @cgbur and @felixscherz thank you for raising this and taking this investigation further. I'm not a polars user myself, but the difference in the behavior is quite interesting, and I think there would be value in trying to fix this issue.
I just read the
write_tableAPI documentation inpyarrow.parquetand found something rather interesting:https://arrow.apache.org/docs/python/generated/pyarrow.parquet.write_table.html#pyarrow.parquet.write_table
If you check the documentation for
use_compliant_nested_typeflag, it mentions that havingelementas the single item field name is the parquet compliant format as specified here on the Parquet Spec for Nested Types. PyArrow defaults to using this flag and writes the list element field name aselement.For some reason it looks like polars has decided to use
item- the non-parquet compliant list element name instead. While I'm curious to know why the polars community has decided to go this route and whether this is common practice in the Parquet community, here are some opinions I've formed on this issue for now:elementoritemname in the function will be very simple, and will allow us to increase our scope of support foradd_filesAPI,itemis not the correct name for the single value field in the LIST field type according to the Parquet Spec. As a result, adding this as a data file to an Iceberg table may have unintended consequences for other parts of our infrastructure that make similar assumptions in accordance to the Parquet Spec. Hence, my hunch is that we should review the potential downstream impact of this change, carefully.cgbur commentedon Jun 20, 2024
Here is the code in Polars where the "item" name crops up.
https://github.com/pola-rs/polars/blob/main/crates/polars-core/src/datatypes/dtype.rs#L575
I believe this is because they are not converting to parquet, but to arrow, then parquet. A subtle difference in their internal logic. However, perhaps the confusion arises because in arrow, the List single element name is often
itemnot element.https://arrow.apache.org/docs/format/Columnar.html#recordbatch-message
I'll go ahead and open an issue on the Polars repo and see if they have anything to say, there are multiple ways to fix this in their package and agree they should likely be producing parquet files according to the spec.
github-actions commentedon Dec 18, 2024
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
github-actions commentedon Jan 1, 2025
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'