Skip to content

Ensure GEOMETRY and GEOGRAPHY tables can be read by pyiceberg - #4422

Draft
paleolimbot wants to merge 5 commits into
apache:mainfrom
paleolimbot:spatial-updates
Draft

paleolimbot wants to merge 5 commits into
apache:mainfrom
paleolimbot:spatial-updates

Conversation

@paleolimbot

Copy link
Copy Markdown
Member

Rationale for this change

Support for geometry and geography is roughed in, but as noted in #4079 , not all of the edge algorithms are mapped and there is at least one correctness issue in the conversion from Iceberg -> GeoArrow. Also, the mechanism by which the types are parsed fails because the CRS is not quoted.

This PR adds one end-to-end read test (Spark writes, PyIceberg reads), and adds the transformation in the other direction (GeoArrow -> PyIceberg) that matches the conversion that pyarrow will do when it writes the Parquet files.

While I was here I simplified the dependency from geoarrow-python to geaorrow-types. I maintain both, and geoarrow-types is pure Python and a better fit for what you're trying to do here.

I can remove any of this...I was just reviewing this in iceberg-rust and thought I'd poke thie along and these all seemed like easy/testable next steps.

Are these changes tested?

Yes

Are there any user-facing changes?

Yes, after this PR GEOMETRY/GEOGRAPHY tables written by Spark (and possibly others) can now be read.

Comment thread pyiceberg/io/pyarrow.py
Comment on lines +1841 to +1850
if (
task.file.file_format == FileFormat.PARQUET
and hasattr(ds.ParquetFragmentScanOptions, "arrow_extensions_enabled")
and any(
isinstance(table_schema.find_type(field_id), (GeometryType, GeographyType))
for field_id in projected_field_ids & table_schema.field_ids
)
):
# Dataset scans disable logical-type extension decoding by default.
format_kwargs["arrow_extensions_enabled"] = True

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure about this part: it seems like it may turn on a switch that had been turned off deliberately (arrow_extensions_enabled is on by default in pyarrow, so I am guessing it was turned off here)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant