Repository navigation
Fix moving a column that was deleted and added back in the same update - #4365
chenjian2664 wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The added-column lookup ignores case_sensitive=False, so moving differently cased replacement columns or using them as anchors can resolve deleted field IDs and raise StopIteration.
1 open issue
What changed in this PR
This PR fixes schema updates that delete a column, add it back under the same name, and then move it by prioritizing newly added columns during move lookup.
Changes:
- Resolve move targets against columns added in the current update before the existing schema.
- Add a regression test for replacing a column’s type and moving it after another column.
| File | Description |
|---|---|
| tests/table/test_init.py | Tests column replacement and reordering within one schema update. |
| pyiceberg/table/update/schema.py | Prioritizes newly added columns when resolving move targets. |
Review effort: Balanced
| if (added_id := self._added_name_to_id.get(name)) is not None: | ||
| return added_id |
Deleting a column, adding it back under the same name and then moving it in one update_schema() raised a bare StopIteration: the move resolved the name to the deleted field. Check the columns added in the update first, and track their names in case-aware form so case_sensitive=False also finds them, as Java's SchemaUpdate.findForMove does.
a42f2a9 to
d9be5f2
Compare
Fokko
left a comment
There was a problem hiding this comment.
Thanks @chenjian2664! Nice find, matches Java's SchemaUpdate.findForMove which also checks added names first. Checked the diff and ran the new tests locally: all pass, and the rest of tests/table/test_init.py is green too (only unrelated env failures).
LGTM!
One tiny nit: the "Are these changes tested?" section is still blank, maybe worth a one-liner since you added two nice tests :)

Rationale for this change
Within one
update_schema(), deleting a column, adding it back under the same name and then moving it withmove_first,move_beforeormove_afterraises a bareStopIteration. This is a natural way to replace a column with an incompatible type while keeping its position. The move resolved the name against the existing schema first and picked up the deleted field.The lookup now checks columns added in the same update first, as Java's
SchemaUpdate.findForMovedoes.Are these changes tested?
Are there any user-facing changes?