Skip to content

Only apply position deletes to the data files they are attached to - #4411

Open
rahulsmahadev wants to merge 2 commits into
apache:mainfrom
rahulsmahadev:rahul/position-delete-readded-file
Open

rahulsmahadev wants to merge 2 commits into
apache:mainfrom
rahulsmahadev:rahul/position-delete-readded-file

Conversation

@rahulsmahadev

Copy link
Copy Markdown
Contributor

Rationale for this change

A position delete file can list rows of several data files. ArrowScan loads the position deletes of every task and
merges them into one map keyed by data file path. A data file then gets the positions of any delete file that names
its path, even when the planner did not attach that delete file to its task. So a delete file that also names another
data file can remove rows from a file it was not planned for, undoing the sequence number and partition checks that
DeleteFileIndex made for it.

The trigger, on a v2 table with a position delete file written by another engine. P has ids 0..9, Q has ids 100..109,
and the delete file, committed before the first line, lists row 0 of P and row 0 of Q:

tbl.delete("id < 10")                       # removes P as a whole file
tbl.add_files([p_path])                     # registers P again, so its data sequence number is newer than the delete's
tbl.scan().to_arrow()                       # 18 rows, id 0 missing; expected 0..9 and 101..109
tbl.scan().count()                          # 19
tbl.scan(row_filter="id < 10").to_arrow()   # 0..9, but the unfiltered scan restricted to id < 10 gives 1..9

The delete predates P, so P must keep its row 0. Scan Planning in format/spec.md says that a position delete applies
only when "The data file's data sequence number is less than or equal to the delete file's data sequence number", and
that the data file and position delete file must have equal partitions (both spec and partition values). Java gives each task only its
own deletes: GenericDeleteFilter is built from task.deletes(), and BaseDeleteLoader reads the delete files it is
given and keeps the positions for the data file path it is asked for.

This change keeps, for each delete file, only the positions of the data files whose task has that delete file attached
(task.delete_files). Each delete file is still read once, and the key is unchanged: the path plus the content offset
and size, so deletion vector ranges in one Puffin file stay separate. The root cause is in the loader, which drops the
association between a delete file and its task, so the lookup in _record_batches_from_scan_tasks_and_deletes is left
as it is. Planning is unchanged; the read path adds one set entry per delete file and task.

_DeleteFileKey comes from #3478. PR #3285 also edits _read_all_delete_files (it skips non-position delete files in
the inner loop over task.delete_files). The two changes merge cleanly, and the merged function keeps both.

Are these changes tested?

Yes. Both new tests fail on main and pass with this change.

  • test_position_delete_respects_sequence_number_of_readded_file (tests/table/test_init.py, catalog fixture) runs the
    trigger above, and a second shape where the delete is committed after P is registered again, so it removes row 0 of
    both files. It asserts the exact rows, that count() agrees with to_arrow(), and that the row-filtered scan agrees
    with the unfiltered scan. On main, the first shape fails because id 0 is missing.
  • test_read_all_delete_files_only_loads_positions_for_tasks_it_is_attached_to (tests/io/test_pyarrow.py) checks the
    loader output for three attachment shapes. The third gives the first data file a newer delete file as well, so a fix
    that merges the positions of all tasks also fails it.

commit_partition_position_delete (tests/table/test_commit_retry.py) now takes a list of (file_path, pos) rows, so one
delete file can name two data files. Its existing callers pass one row.

Are there any user-facing changes?

Yes, a read bug fix: scans no longer drop rows that a newer data file keeps, and an upsert over such a file no longer
inserts a duplicate of a row it should update. There is no API change.

Please add the changelog label.

This pull request and its description were written by Isaac.

_read_all_delete_files merged the positions of every position delete file in a scan by data file path, so a
delete file that also names another data file removed that file's rows, even when the planner had not attached
the delete to it (for example an older delete for a file that add_files registered again). Keep, per delete
file, only the positions of the data files whose task has it attached, reading each delete file once.

Co-authored-by: Isaac <no-reply@databricks.com>
@rahulsmahadev

Copy link
Copy Markdown
Contributor Author

@geruh, this keeps each position delete to the data files that DeleteFileIndex (#2918) attached it to. @KaiqiJinWow, you last changed _read_all_delete_files in #3478. Could you take a look when you have a chance?

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant