Skip to content

Add update_partition_statistics() to Table and Transaction - #4397

Merged
Fokko merged 2 commits into
apache:mainfrom
rahulsmahadev:rahul/update-partition-statistics
Nov 18, 2026
Merged

Fokko merged 2 commits into
apache:mainfrom
rahulsmahadev:rahul/update-partition-statistics

Conversation

@rahulsmahadev

@rahulsmahadev rahulsmahadev commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

PyIceberg already models partition statistics in the table metadata: PartitionStatisticsFile, TableMetadata.partition_statistics, and the SetPartitionStatisticsUpdate and RemovePartitionStatisticsUpdate table updates (#2192). Table and Transaction have no builder for these updates; callers currently need to construct the update objects and commit them through the catalog API.

Java has this API. Table.updatePartitionStatistics() and Transaction.updatePartitionStatistics() return an UpdatePartitionStatistics with setPartitionStatistics(file) and removePartitionStatistics(snapshotId) (api/src/main/java/org/apache/iceberg/UpdatePartitionStatistics.java, core/src/main/java/org/apache/iceberg/SetPartitionStatistics.java).

This PR adds the same API to PyIceberg:

  • Table.update_partition_statistics() and Transaction.update_partition_statistics(), next to update_statistics().
  • UpdatePartitionStatistics in pyiceberg/table/update/partition_statistics.py, modelled on UpdateStatistics. Pending operations are applied on commit, for chained and with block operations alike, with sets applied before removals as in Java.
  • set_partition_statistics(snapshot_id, statistics_path, file_size_in_bytes) builds the PartitionStatisticsFile and replaces the file already set for that snapshot. It takes the fields rather than the file object, so PartitionStatisticsFile can evolve separately from this API. Java's setPartitionStatistics takes the file object.
  • remove_partition_statistics(snapshot_id) uses the existing RemovePartitionStatisticsUpdate.
table.update_partition_statistics().set_partition_statistics(
    snapshot_id=snapshot_id, statistics_path="s3://bucket/partition-stats.parquet", file_size_in_bytes=1024
).commit()

with table.update_partition_statistics() as update:
    update.set_partition_statistics(
        snapshot_id=snapshot_id, statistics_path="s3://bucket/partition-stats.parquet", file_size_in_bytes=1024
    )
    update.remove_partition_statistics(other_snapshot_id)

Related open PRs: #4087 fixes UpdateStatistics.remove_statistics, which replaces the pending operations instead of appending to them. This PR does not copy that pattern. #4364 removes partition statistics when their snapshots are removed. This PR changes only the RemovePartitionStatisticsUpdate handler in pyiceberg/table/update/__init__.py, a different part of that file from the snapshot removal handler that #4364 changes, and does not touch the snapshot expiry code. #4364 only removes entries that exist, so the no-op change does not affect its path.

Are these changes tested?

Yes. Four tests in tests/table/test_statistics.py use the catalog fixture (in-memory and SQL catalogs) and go through the public API:

  • test_update_partition_statistics: set, replace the file for the same snapshot, set in a with block, remove, and reload the table from the catalog.
  • test_update_partition_statistics_set_and_remove_in_one_update: operations on different snapshots are both applied. On the same snapshot the removal wins in either call order, with or without an existing file. Repeated sets keep the last file, and repeated removals are no-ops.
  • test_update_partition_statistics_in_transaction: pending changes are visible in transaction.table_metadata, but not in table.metadata or a catalog reload, until the transaction commits. An exception rolls them back.
  • test_remove_partition_statistics_without_file_is_noop: removing a snapshot without a file, once or repeated, commits and leaves the metadata unchanged.

The low-level handler is covered by test_remove_partition_statistics_update_without_file_is_noop in tests/table/test_init.py.

All four tests fail on main with AttributeError: the transaction test reports that Transaction has no update_partition_statistics attribute, and the other three report the missing attribute on Table. All four pass with this change. The low-level test in tests/table/test_init.py fails on main with ValueError and passes with this change. I also checked that a copy of the UpdateStatistics pattern, where removal discards the pending sets, fails the set-and-remove test.

Are there any user-facing changes?

Yes. New methods Table.update_partition_statistics() and Transaction.update_partition_statistics(), and a new UpdatePartitionStatistics class. The docs are in mkdocs/docs/api.md. Two behaviours follow Java:

  • Removing partition statistics that are absent is a no-op, matching Java. This also changes the existing RemovePartitionStatisticsUpdate handler, which previously raised ValueError.
  • Within a builder, sets are applied before removals, matching Java. Removing a snapshot's statistics wins over setting them, regardless of call order.

Please add the changelog label.

This pull request and its description were written by Isaac.

Add an UpdatePartitionStatistics builder with set_partition_statistics() and
remove_partition_statistics(), so callers no longer build update objects by
hand. Sets apply before removals and absent removals are no-ops, as in Java.
The RemovePartitionStatisticsUpdate handler therefore no longer raises.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@rambleraptor, this adds Table and Transaction builders for the partition-statistics updates you introduced in #2192. Could you take a look at the API, and at the missing-file and operation-order semantics, when you have a chance?

@Fokko

Fokko commented Nov 17, 2026

Copy link
Copy Markdown
Contributor

Zooming out a bit more, wouldn't it make more sense to have PyIceberg compute the partition statistics?

@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

Good point. Java splits the two: PartitionStatsHandler.computeAndWriteStatsFile computes the stats (incrementally from the previous stats file) and writes the file, and ComputePartitionStatsSparkAction then registers it with table.updatePartitionStatistics().setPartitionStatistics(file).commit(). This PR adds that registration step, so a compute step can build on it. I'm happy to follow up with a PR that computes and writes the partition statistics file in PyIceberg and registers it through this API, or to add the computation here if you'd rather review them together. Which do you prefer?

self._sets = {}
self._removals = {}

def set_partition_statistics(self, partition_statistics_file: PartitionStatisticsFile | None) -> "UpdatePartitionStatistics":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's avoid passing in the PartitionStatisticsFile itself. Let's pass in the parameters to construct a PartitionStatisticsFile instead, so the object can evolve separately from the API here.

Also, let's make sure to add docstrings here and at remove_partition_statistics, since it is a public API

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 964b6f0. set_partition_statistics now takes snapshot_id, statistics_path and file_size_in_bytes and builds the PartitionStatisticsFile itself, and both public methods have docstrings. The docs and tests use the new signature; I dropped the None no-op, since there is no file argument any more.

Review feedback: set_partition_statistics takes snapshot_id, statistics_path
and file_size_in_bytes and builds the PartitionStatisticsFile, so the model
can evolve separately from the API. Add docstrings to both public methods.

Co-authored-by: Isaac <no-reply@databricks.com>

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rahulsmahadev for working on this 🚀

@Fokko
Fokko added this pull request to the merge queue Nov 18, 2026
Merged via the queue into apache:main with commit e1fc90e Nov 18, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants