Skip to content

fix(inspect): map partition record by field id to support renamed partition fields - #4363

Open
Ankitkj1999 wants to merge 1 commit into
apache:mainfrom
Ankitkj1999:fix/inspect-renamed-partition-field
Open

Ankitkj1999 wants to merge 1 commit into
apache:mainfrom
Ankitkj1999:fix/inspect-renamed-partition-field

Conversation

@Ankitkj1999

Copy link
Copy Markdown

Closes #4362

Rationale for this change

When a partition field is renamed (via update_spec().rename_field() or via v1 auto-renaming voided fields on re-add), specs_struct() adopts the latest field name per field_id. Previously, inspect.py constructed partition dictionaries using each entry's historical spec field names directly (field.name), causing PyArrow conversion to silently drop pre-rename partition values as None (or misattribute them to newly added fields sharing the old name).

This PR:

  1. Updates InspectTable.entries(), partitions(), and _get_files_from_manifest() to map partition values by field.field_id through specs_struct(), preserving historical partition values in the PyArrow output.
  2. Keys partitions_map in partitions() by (file.spec_id, partition_record_dict) so partition counts and statistics are tracked per specification layout.

Are these changes tested?

Yes:

  • Added test_inspect_partitions_and_files_after_rename_field in tests/table/test_inspect.py testing partitions(), files(), data_files(), all_files(), all_data_files(), and entries().
  • Added test_inspect_files_after_v1_partition_field_readd in tests/table/test_inspect.py testing field attribution across voided and re-added partition fields in format v1 tables.

Are there any user-facing changes?

Bugfix: table.inspect tables accurately report historical partition values after partition field renames and partition re-additions.

…tition fields

When a partition field is renamed (via update_spec().rename_field() or via v1 auto-renaming voided fields on re-add), specs_struct() adopts the latest field names per field_id. Previously, inspect.py constructed partition dicts using the historical spec's field names directly, causing PyArrow conversion to silently drop pre-rename partition values as None (or misattribute them to newly added fields sharing the old name).

This updates InspectTable.entries(), partitions(), and _get_files_from_manifest() to map partition field values by field_id using specs_struct() names, and keys partitions_map by (spec_id, partition_record_dict) to track partition stats per specification layout.
@Ankitkj1999
Ankitkj1999 force-pushed the fix/inspect-renamed-partition-field branch from d8cf7b2 to 50939d9 Compare November 13, 2026 06:48

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

table.inspect metadata tables silently drop (or misattribute) partition values after a partition field rename

1 participant