Repository navigation
Only apply position deletes to the data files they are attached to - #4411
Open
rahulsmahadev wants to merge 2 commits into
Open
rahulsmahadev wants to merge 2 commits into
rahulsmahadev wants to merge 2 commits into
Conversation
_read_all_delete_files merged the positions of every position delete file in a scan by data file path, so a delete file that also names another data file removed that file's rows, even when the planner had not attached the delete to it (for example an older delete for a file that add_files registered again). Keep, per delete file, only the positions of the data files whose task has it attached, reading each delete file once. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
@geruh, this keeps each position delete to the data files that DeleteFileIndex (#2918) attached it to. @KaiqiJinWow, you last changed |
Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
A position delete file can list rows of several data files.
ArrowScanloads the position deletes of every task andmerges them into one map keyed by data file path. A data file then gets the positions of any delete file that names
its path, even when the planner did not attach that delete file to its task. So a delete file that also names another
data file can remove rows from a file it was not planned for, undoing the sequence number and partition checks that
DeleteFileIndexmade for it.The trigger, on a v2 table with a position delete file written by another engine. P has ids 0..9, Q has ids 100..109,
and the delete file, committed before the first line, lists row 0 of P and row 0 of Q:
The delete predates P, so P must keep its row 0. Scan Planning in
format/spec.mdsays that a position delete appliesonly when "The data file's data sequence number is less than or equal to the delete file's data sequence number", and
that the data file and position delete file must have equal partitions (both spec and partition values). Java gives each task only its
own deletes:
GenericDeleteFilteris built fromtask.deletes(), andBaseDeleteLoaderreads the delete files it isgiven and keeps the positions for the data file path it is asked for.
This change keeps, for each delete file, only the positions of the data files whose task has that delete file attached
(
task.delete_files). Each delete file is still read once, and the key is unchanged: the path plus the content offsetand size, so deletion vector ranges in one Puffin file stay separate. The root cause is in the loader, which drops the
association between a delete file and its task, so the lookup in
_record_batches_from_scan_tasks_and_deletesis leftas it is. Planning is unchanged; the read path adds one set entry per delete file and task.
_DeleteFileKeycomes from #3478. PR #3285 also edits_read_all_delete_files(it skips non-position delete files inthe inner loop over
task.delete_files). The two changes merge cleanly, and the merged function keeps both.Are these changes tested?
Yes. Both new tests fail on main and pass with this change.
test_position_delete_respects_sequence_number_of_readded_file(tests/table/test_init.py,catalogfixture) runs thetrigger above, and a second shape where the delete is committed after P is registered again, so it removes row 0 of
both files. It asserts the exact rows, that
count()agrees withto_arrow(), and that the row-filtered scan agreeswith the unfiltered scan. On main, the first shape fails because id 0 is missing.
test_read_all_delete_files_only_loads_positions_for_tasks_it_is_attached_to(tests/io/test_pyarrow.py) checks theloader output for three attachment shapes. The third gives the first data file a newer delete file as well, so a fix
that merges the positions of all tasks also fails it.
commit_partition_position_delete(tests/table/test_commit_retry.py) now takes a list of(file_path, pos)rows, so onedelete file can name two data files. Its existing callers pass one row.
Are there any user-facing changes?
Yes, a read bug fix: scans no longer drop rows that a newer data file keeps, and an upsert over such a file no longer
inserts a duplicate of a row it should update. There is no API change.
Please add the
changeloglabel.This pull request and its description were written by Isaac.