Skip to content

Fail a retried upsert instead of inserting a concurrently inserted key - #4392

Open
Anurag-S1ngh wants to merge 2 commits into
apache:mainfrom
Anurag-S1ngh:fix/4194-upsert-retry-concurrent-insert
Open

Anurag-S1ngh wants to merge 2 commits into
apache:mainfrom
Anurag-S1ngh:fix/4194-upsert-retry-concurrent-insert

Conversation

@Anurag-S1ngh

Copy link
Copy Markdown
Contributor

Summary

A commit retry replays the append planned before the concurrent commit. The fast append producer skips concurrency validation, so a key another writer inserted after the upsert read was inserted a second time.

Changes

  • Register the filter for the rows an upsert inserts on the append producer.
  • On retry, validate the data files added since the starting snapshot against that filter and raise CommitFailedException instead of inserting a duplicate key.
  • Plain appends keep skipping validation.

Testing

  • Added a test with two table handles where one upserts a key after the other has read the table, asserting the second upsert fails and the key is stored once.
  • make test passes (4697 passed, 5 skipped).

Closes #4194

A commit retry replays the append that was planned before the concurrent
commit, and the fast append producer skips concurrency validation, so a key
another writer inserted after the upsert read ends up inserted twice.

Record the filter for the rows an upsert inserts on the append producer and
validate the files added since the starting snapshot against it when the
commit is rebuilt for a retry. The upsert now fails with a
CommitFailedException instead of leaving a duplicate key.

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for picking this up, the approach of validating the added data files against the inserted keys on retry looks right to me. A few comments inline.

self._upsert_insert_filter = insert_filter
self._case_sensitive = case_sensitive

def _validate_concurrency(self) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check runs regardless of the table's isolation level. The update half of the upsert only runs _validate_added_data_files under serializable (see the base _validate_concurrency), so with write.update.isolation-level=snapshot the two halves of a single upsert behave differently: the update half tolerates concurrently added files, while the insert half rejects them. This matches Java, where MERGE only validates conflicting data under serializable isolation. Should we check isolation_level(Operation.OVERWRITE) here so both halves follow the same setting?

Comment thread pyiceberg/table/update/snapshot.py Outdated
self._case_sensitive,
)
except ValidationException as e:
raise CommitFailedException(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A conflict on the update half of the upsert raises ValidationException, but here we convert it into a CommitFailedException. This means that the same upsert raises a different exception depending on which half conflicted. I can see the argument for CommitFailedException (it's what 0.11.1 raised, and the caller can simply retry the whole upsert), but I think we should be consistent across both halves. What do you think about letting the ValidationException propagate?

Comment thread pyiceberg/table/__init__.py Outdated
df: pa.Table | pa.RecordBatchReader,
snapshot_properties: dict[str, str] = EMPTY_DICT,
branch: str | None = MAIN_BRANCH,
_upsert_insert_filter: BooleanExpression | None = None,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not a big fan of adding private _upsert_insert_filter / _case_sensitive arguments to the public Transaction.append signature, since they show up in the API docs. Could upsert configure the producer directly instead? Also, the concept is not specific to upsert; it's a conflict detection filter. Something like _conflict_detection_filter / validate_no_conflicting_data(filter) would line up with Java's OverwriteFiles/RowDelta. Last, set_upsert_insert_filter also overwrites the inherited _case_sensitive, which is harmless today but easy to trip over later.

Address the review feedback on the upsert retry check:

- Gate the check on write.update.isolation-level, so both halves of an upsert
  follow the same isolation setting.
- Let the ValidationException propagate instead of converting it to a
  CommitFailedException, matching the update half.
- Drop the private Transaction.append arguments and let upsert configure the
  append producer through validate_no_conflicting_data. The filter's case
  sensitivity is stored on the producer instead of overwriting its inherited
  value.
@Fokko
Fokko self-requested a review November 17, 2026 20:46

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Anurag-S1ngh, nice catch and a clean fix! The second commit addressing the isolation level gating and letting the ValidationException propagate reads well.

I pulled the branch and ran tests/table/test_upsert.py locally, all 29 pass, and CI is green.

A couple of small things, none blocking:

  • pyiceberg/table/update/snapshot.py:695: validate_no_conflicting_data only stores the filter, the actual check runs in _validate_concurrency on retry. Maybe worth a docstring line saying the validation happens at commit/retry time, so callers don't expect an exception from the call itself.
  • pyiceberg/table/update/snapshot.py:693: the class level defaults on _FastAppendFiles mean a producer that never calls validate_no_conflicting_data keeps skipping validation, which is what plain appends want. Good.
  • The snapshot isolation test documenting that the duplicate can still land under write.update.isolation-level=snapshot is a nice touch, that makes the semantics explicit.

LGTM once CI stays green!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A retried upsert inserts a key that a concurrent writer already inserted

2 participants