Skip to content

Write Parquet page indexes by default - #4402

Open
Anurag-S1ngh wants to merge 5 commits into
apache:mainfrom
Anurag-S1ngh:feat/3847-parquet-page-index
Open

Anurag-S1ngh wants to merge 5 commits into
apache:mainfrom
Anurag-S1ngh:feat/3847-parquet-page-index

Conversation

@Anurag-S1ngh

@Anurag-S1ngh Anurag-S1ngh commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

PyIceberg never wrote Parquet page indexes, so readers that support them cannot skip non-matching pages during predicate evaluation. PyArrow already supports this with ParquetWriter(write_page_index=True), and the Java writer writes these indexes by default, so PyIceberg now does the same.

Changes

  • Write Parquet column and offset indexes by default, matching the Java writer.
  • Add the write.parquet.page-index-enabled table property, default true, as an opt-out.
  • Document the property in the write options table.
  • Update the expected column sizes in the manifest merge integration test, since files written with page indexes have different column chunk sizes.

Testing

  • Added a test that writes with the property defaulting to on and explicitly set to false, and checks has_column_index and has_offset_index on the resulting row group.
  • Unit tests, lint, and integration tests run in CI.

Closes #3847

Readers that support page indexes can skip non-matching pages during
predicate evaluation, but PyIceberg never wrote them. Add the
write.parquet.page-index-enabled table property, off by default so the
current output and file sizes are preserved, and pass it to the Parquet
writer.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two things:

  1. I'm not sure if we need this property. It is very specific, and Java doesn't have it either.
  2. Java writes the indexes by default, so I feel like we should also do that in PyIceberg

@Anurag-S1ngh

Copy link
Copy Markdown
Contributor Author

Yes, my mistake, I have checked. I've flipped the default so PyIceberg writes the column and offset indexes by default too, matching Java. I kept write.parquet.page-index-enabled as an opt-out in case someone needs to disable it, but I can drop the property entirely if you'd rather have strict parity.

@Anurag-S1ngh Anurag-S1ngh changed the title Add an opt-in property to write Parquet page indexes Write Parquet page indexes by default Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support optional Parquet page index writes

2 participants