Skip to content

feat: proto serialization for PiecewiseMergeJoinExec - #24378

Open
SubhamSinghal wants to merge 2 commits into
apache:mainfrom
SubhamSinghal:pwmj-proto-serialization
Open

feat: proto serialization for PiecewiseMergeJoinExec#24378
SubhamSinghal wants to merge 2 commits into
apache:mainfrom
SubhamSinghal:pwmj-proto-serialization

Conversation

@SubhamSinghal

@SubhamSinghal SubhamSinghal commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #17427.

Rationale for this change

PiecewiseMergeJoinExec has no protobuf representation, so a query the planner turns into a range join cannot be serialized and cannot run on any engine that ships physical plans between processes (Ballista, Flight-based executors):

Internal("Unsupported plan and extension codec failed with
  [This feature is not implemented: PhysicalExtensionCodec is not provided].
  Plan: PiecewiseMergeJoinExec { ... }")

What changes are included in this PR?

  • New PiecewiseMergeJoinExecNode at oneof PhysicalPlanType tag 40, carrying exactly the six arguments of try_new. Everything else (output schema, sort_options, required input orderings, PlanProperties) is derived inside try_new, so it is not on the wire and the round trip is exact by construction.
  • ExecutionPlan::try_to_proto + PiecewiseMergeJoinExec::try_from_proto, following the self-serialization pattern ([EPIC] Apply try_to_proto / try_from_proto pattern to ExecutionPlan #22419) that SortMergeJoinExec uses. Since the hook is consulted before the central downcast chain, the encode side needs no change in datafusion/proto; only one decode arm is added.
  • Regenerated pbjson.rs / prost.rs via datafusion/proto-models/regen.sh.

Are these changes tested?

Yes using UT

Are there any user-facing changes?

No breaking changes

@github-actions github-actions Bot added proto Related to proto crate physical-plan Changes to the physical-plan crate labels Aug 14, 2026
@SubhamSinghal
SubhamSinghal marked this pull request as ready for review August 14, 2026 17:30
@github-actions

github-actions Bot commented Aug 14, 2026 •
edited
Loading

Copy link
Copy Markdown

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details 
     Cloning apache/main
    Building datafusion-physical-plan v54.1.0 (current)
       Built [  38.460s] (current)
     Parsing datafusion-physical-plan v54.1.0 (current)
      Parsed [   0.150s] (current)
    Building datafusion-physical-plan v54.1.0 (baseline)
       Built [  37.705s] (baseline)
     Parsing datafusion-physical-plan v54.1.0 (baseline)
      Parsed [   0.152s] (baseline)
    Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   1.048s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [  78.953s] datafusion-physical-plan
    Building datafusion-proto v54.1.0 (current)
       Built [  54.701s] (current)
     Parsing datafusion-proto v54.1.0 (current)
      Parsed [   0.021s] (current)
    Building datafusion-proto v54.1.0 (baseline)
       Built [  53.474s] (baseline)
     Parsing datafusion-proto v54.1.0 (baseline)
      Parsed [   0.021s] (baseline)
    Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   0.163s] 223 checks: 223 pass, 31 skip
     Summary no semver update required
    Finished [ 109.626s] datafusion-proto
    Building datafusion-proto-models v54.1.0 (current)
       Built [  24.903s] (current)
     Parsing datafusion-proto-models v54.1.0 (current)
      Parsed [   0.137s] (current)
    Building datafusion-proto-models v54.1.0 (baseline)
       Built [  24.770s] (baseline)
     Parsing datafusion-proto-models v54.1.0 (baseline)
      Parsed [   0.136s] (baseline)
    Checking datafusion-proto-models v54.1.0 -> v54.1.0 (no change; assume patch)
     Checked [   2.534s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip

--- failure enum_variant_added: enum variant added on exhaustive enum ---

Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
        ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron

Failed in:
  variant PhysicalPlanType:PiecewiseMergeJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1432
  variant PhysicalPlanType:PiecewiseMergeJoin in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:1432

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  54.062s] datafusion-proto-models

@github-actions github-actions Bot added the auto detected api change Auto detected API change label Aug 14, 2026
@codecov-commenter

codecov-commenter commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 32.07547% with 144 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.16%. Comparing base (7c079f7) to head (6e8466f).

Files with missing lines Patch % Lines
datafusion/proto-models/src/generated/pbjson.rs 0.00% 137 Missing ⚠️
...ysical-plan/src/joins/piecewise_merge_join/exec.rs 90.54% 2 Missing and 5 partials ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main   #24378      +/-   ##
==========================================
- Coverage   81.19%   81.16%   -0.03%     
==========================================
  Files        1110     1110              
  Lines      388739   388951     +212     
  Branches   388739   388951     +212     
==========================================
+ Hits       315636   315702      +66     
- Misses      54515    54656     +141     
- Partials    18588    18593       +5

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

auto detected api change Auto detected API change physical-plan Changes to the physical-plan crate proto Related to proto crate

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@SubhamSinghal @codecov-commenter