Skip to content

table.inspect metadata tables fail (or lose manifest bounds) after dropping an old partition column #4355

Open

Description

@Ankitkj1999

Apache Iceberg version

main (development)

Please describe the bug 🐞

Describe the bug

If a table stops partitioning by a column (update_spec().remove_field(...)) and later drops that source column (update_schema().delete_column(...)):

  1. Calling table.inspect.partitions(), entries(), files(), data_files(), delete_files(), all_files(), all_data_files(), or all_delete_files() raises ValueError: Could not find field with id: ... inside TableMetadata.specs_struct().
  2. Calling table.inspect.manifests() / all_manifests() succeeds, but renders lower_bound and upper_bound in partition_summaries as 'null' because _partition_summaries_to_rows() evaluates spec.partition_type(self.tbl.schema()), which falls back to UnknownType().

(Note: open PR #4115 fixes delete() manifest rewrites in pyiceberg/avro/file.py and pyiceberg/manifest.py, whereas this failure occurs in TableMetadata.specs_struct() and InspectTable even without calling delete().)

To Reproduce

import tempfile
import pyarrow as pa
from pyiceberg.catalog.memory import InMemoryCatalog
from pyiceberg.partitioning import PartitionField, PartitionSpec
from pyiceberg.schema import Schema
from pyiceberg.transforms import IdentityTransform
from pyiceberg.types import LongType, NestedField, StringType

with tempfile.TemporaryDirectory() as tmp:
    cat = InMemoryCatalog("local", warehouse=f"file://{tmp}/wh")
    cat.create_namespace("ns")
    tbl = cat.create_table(
        "ns.t_v2",
        schema=Schema(NestedField(1, "id", LongType()), NestedField(2, "p", StringType())),
        partition_spec=PartitionSpec(PartitionField(2, 1000, IdentityTransform(), "p")),
        properties={"format-version": "2"},
    )
    tbl.append(pa.table({"id": [1, 2], "p": ["a", "b"]}))
    with tbl.update_spec() as spec_update:
        spec_update.remove_field("p")
    with tbl.update_schema() as schema_update:
        schema_update.delete_column("p")

    print(tbl.inspect.manifests().to_pydict()["partition_summaries"])
    # [[{'contains_null': False, 'contains_nan': False, 'lower_bound': 'null', 'upper_bound': 'null'}]]

    tbl.inspect.partitions()
    # ValueError: Could not find field with id: 2

Expected behavior

TableMetadata.specs_struct() and table.inspect.* should resolve partition source field types from historical table schemas when the source column was dropped from the current schema, keeping metadata inspection tables and manifest partition summary bounds readable.

Willingness to contribute

  • I can contribute a fix for this bug independently
  • I would be willing to contribute a fix for this bug with guidance from the Iceberg community
  • I cannot contribute a fix for this bug at this time
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions