[WIP][SPARK-57378][SDP] Implement SCD2 Batch Processor; Merge Reconciled Rows into Aux and Target Tables#57444
Open
anew wants to merge 5 commits into
Open
[WIP][SPARK-57378][SDP] Implement SCD2 Batch Processor; Merge Reconciled Rows into Aux and Target Tables#57444anew wants to merge 5 commits into
anew wants to merge 5 commits into
Conversation
(cherry picked from commit eac615f)
(cherry picked from commit 48f6408)
Cleanups on the merge-into-aux-and-target implementation:
- Remove 4 trailing-whitespace lines flagged by scalastyle in
mergeRowsIntoAuxiliaryTable.
- Fix typo auxTableDeleyedByBatchId -> auxTableDeletedByBatchId.
- Rename misleading auxTableRecordStartAt to targetTableRecordStartAt in
mergeRowsIntoTargetTable (it reads from the target, not the aux, table).
- Fix grammar in the isVisibleTargetRow comment ("its" -> "it's",
"Only op upsert" -> "Only non-no-op upsert").
- Turn the aux-table GC full-scan caveat into an explicit TODO.
Fill coverage gaps in the merge/tag paths: - identifyAndTagAuxRows: a two-key case asserting the per-key window never lets one key's run influence another's tagging. - identifyAndTagAuxRows / mergeRowsIntoAuxiliaryTable / mergeRowsIntoTargetTable: user columns whose names contain a dot (e.g. `user.name`) must be quoted at every pass-through/assignment site, matching the coverage reconcileStartAndEndAt already has. Each test fails with UNRESOLVED_COLUMN if the corresponding QuotingUtils.quoteIdentifier call is dropped.
Fill the remaining merge coverage gaps: - mergeRowsIntoAuxiliaryTable: one merge that simultaneously inserts a new routed row, updates a surviving row, logically deletes a non-surviving affected row, and garbage-collects an older-batch row - exercising the whenMatched/whenNotMatched/whenNotMatchedBySource clauses together. - mergeRowsIntoAuxiliaryTable and mergeRowsIntoTargetTable: an in-place update over a schema with several non-key columns (name, status, score), asserting every non-key column is refreshed - guards the non-key update assignment map against dropping columns. Each test fails under the corresponding mutation (GC clause removed, or the assignment map truncated to one column).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Note: This continues @AnishMahto's #56457, opened from my fork while the original author is out of office. It carries his implement merge into aux and target + self-review commits and adds review fixes and additional test coverage on top. #56457 should be closed in favor of this PR to keep review in one place.
What changes were proposed in this pull request?
This is the next step in the SCD2 AutoCDC batch-processor series (following SPARK-57356, "Cleanup Delete Encoding Rows Post-Reconciliation"). It adds the final merge stage that writes a reconciled microbatch into the auxiliary and target tables. Three methods are added to Scd2BatchProcessor:
Together with the reconciliation and cleanup steps already merged, this completes the per-microbatch transformation pipeline; the foreachBatch callback that drives it end-to-end lands in the follow-up (SPARK-57395).
Why are the changes needed?
The SCD2 batch processor is being built incrementally across the SPARK-571xx series. After a microbatch is preprocessed, decomposed, reconciled, and cleaned up, its rows must actually be written to the auxiliary and target tables using the correct MERGE semantics. This PR supplies that final merge step, which the engine's SCD2 execution path depends on.
Does this PR introduce any user-facing change?
No. These are internal (private[autocdc]) batch-processor methods; SCD2 AutoCDC is not yet reachable end-to-end (the driving foreachBatch handler and the API/planner wiring land in separate changes).
How was this patch tested?
New unit tests:
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)