Repository navigation
Upgrade DataFusion to version 56 - #1826
nuno-faria wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
There was a problem hiding this comment.
There was a problem hiding this comment.
| } | ||
|
|
||
| /// Retrieves the right input `LogicalPlan` to this `Join` node | ||
| /// Retrieves the equality keys of this `Join` node |
There was a problem hiding this comment.
Noticed when creating the ASOF join that the prev comment was wrong.
There was a problem hiding this comment.
| &self, | ||
| _ctx: &dyn Session, | ||
| projection: Option<&Vec<usize>>, | ||
| projection: Option<&[usize]>, |
There was a problem hiding this comment.
| @@ -5385,7 +5385,7 @@ def map_entries(map: Expr) -> Expr: | |||
| def element_at(map: Expr, key: Expr) -> Expr: | |||
| """Returns the value for a given key in the map. | |||
|
|
|||
| Returns ``[None]`` if the key is absent. | |||
| Returns ``[]`` if the key is absent. | |||
There was a problem hiding this comment.
|
|
||
| def test_sql_with_options_no_statements(ctx): | ||
| sql = "SET time zone = 1;" | ||
| sql = "SET time zone = 'UTC';" |
There was a problem hiding this comment.
Timezone is now validated.
| @@ -1811,7 +1811,7 @@ def test_read_csv_with_options(tmp_path, as_read, global_ctx): | |||
| [ | |||
| pa.array(["Alice", "Bob", "Phil", "Karin"]), | |||
| pa.array([30.0, 25.0, 75.0, 50.0]), | |||
| pa.array(["New York; NY", None, "Detroit' MI", "Stockholm\nSweden"]), | |||
| pa.array(["New York; NY", "", "Detroit' MI", "Stockholm\nSweden"]), | |||
There was a problem hiding this comment.
This now appears as an empty string since it does not match the null_regex. From apache/datafusion#25254
| with pytest.raises( | ||
| Exception, match=r"Invalid compression level \d+ for \w+\. Expected a level" | ||
| ): |
There was a problem hiding this comment.
|
@timsaucer I think the problem with the _test_three_library_query_planner comes from the FFI crate upstream: It appears that the problem is now shown by apache/datafusion#26461. |
|
@nuno-faria you were right that the
I cherry-picked the fix onto the |
andygrove
left a comment
There was a problem hiding this comment.
This is an AI generated review. Feel free to ignore any points that you do not agree with.
Blocking
test_spawning_plan_across_three_librariesfails because of an upstream FFI bug.FFI_ExecutionPlandoesn't carry input requirements across the boundary. Tracked in apache/datafusion#27427, with a fix in apache/datafusion#27428. With the fix applied tobranch-56, this PR passes all FFI example tests (51/51 and 59/59).- CI's TPC-H step will fail next.
tpchgen-cli4.0 removed--format. Here the step was skipped only because the FFI step failed first. #1832 fixes it, and this PR will need to rebase onto it. - Dependency source.
[patch.crates-io]points at a moving gitbranch-56, and the comment above it says to remove it before a release candidate. For now it could bump to the currentbranch-56tip (cce6374e6, which includes the 56.0.0 version bump). I ran the full suite against that tip with the FFI fix applied, and it is clean: main suite with doctests, both FFI example suites, TPC-H 22/22 and clippy. Once 56.0.0 is on crates.io it should switch to that.
Should fix
Show.schema()returnsNoneforDESCRIBE <table>(crates/core/src/expr/show.rs). PreviouslyDescribeTable.schema()returned the table's schema in this case. Part of this comes from upstream:ShowKind::Columnsnow holds onlytable_refanddetail, with no schema. But the binding exposes notable_refand none of the other kinds' payloads (variable name, function filter, etc.), so users can't get the table back. The upgrade guide showskind() == "columns"but doesn't mention thatschema()is nowNone. It could suggestctx.table(name).schema()instead.- Some user-visible behavior changes are missing from
upgrade-guides.md. The tests and docstrings were updated for these, but the guide wasn't:map_extract/element_atwith a missing key now returns[]instead of[None].- CSV
null_regexnow also applies to the data, not only to schema inference (apache/datafusion#25254). Fields that don't match the regex, including the empty fields of truncated rows, come back as""instead of null. This looks like intended upstream behavior and the test update is correct, but it is worth a note. - Time zone values in
SETare now validated.
- The new bindings aren't covered from Python.
test_asof_joinandtest_materialized_cteonly comparedisplay_indent()output, which is upstream's RustDisplay. No test reachesAsOfJoin,MaterializedCte,MaterializedCteScan,CreateExternalCatalogorDropCatalogthroughto_variant()or their accessors (match_condition(),on(), etc.). A wrong match arm or a broken accessor inPyLogicalPlan::to_variantwould still pass. crates/core/src/expr/describe_table.rsis orphaned. Itsmodline was removed, but the file is still tracked and still references the removed upstreamDescribeTable. It can be deleted.- MSRV. The workspace has
rust-version = "1.88", but DataFusion 56 declaresrust-version = "1.95.0"(andrust-toolchain.tomlis now 1.99.0). It should be raised to at least 1.95.
Minor
- The Python pin also applies to Windows. In
.github/workflows/build.yml,uv sync --python 3.14is in the job whose matrix isos: [macos-latest, windows-latest], so the Windows abi3 build is pinned too, while the commit message says macOS only. The pin is also repeated in two jobs. Show(schema, output_schema)can only buildShowKind::Describe, andkind()returns bare strings. That's fine for now, but a docstring note listing the possiblekind()values would help users who match on them.
## Which issue does this PR close? - Closes apache#27427. ## Rationale for this change A `ForeignExecutionPlan` reports the `ExecutionPlan` defaults for its input requirements (`UnspecifiedDistribution`, no required ordering, `maintains_input_order = false`), whatever the wrapped plan requires. When a physical optimizer rule runs on the other side of the FFI boundary from the plan it optimizes, it can't see those requirements. That happens with a foreign `QueryPlanner` whose session rules belong to the host. `EnsureRequirements` then puts only a round-robin repartition under a `FinalPartitioned` aggregate, which can split one group key across several partitions. For FFI query planner users this is a regression from 55 to 56: a query that works on 55.0.0 fails at planning time on 56. Until apache#26461, `SanityCheckPlan` ran as a host-side rule and was just as blind, so the plan got through (correct by luck in the affected test, unsafe in general). Since apache#26461 the check runs inside `DefaultPhysicalPlanner` and rejects it. This is the only failure in the datafusion-python upgrade to 56 (apache/datafusion-python#1826). The issue has the full error, the plan and the 55 → 56 history. ## What changes are included in this PR? - `FFI_ExecutionPlan` gets four callbacks: `input_distribution_requirements`, `required_input_ordering`, `maintains_input_order` and `benefits_from_input_partitioning`. `ForeignExecutionPlan` overrides the matching methods, and also the deprecated `required_input_distribution` so it agrees. - `ForeignExecutionPlan` fetches the requirements once in `TryFrom<FFI_ExecutionPlan>`, the way it already does `properties`, instead of crossing FFI on every optimizer call. A malformed ordering requirement fails the conversion rather than being dropped. - `FFI_PhysicalExpr` gets an `eq` callback, so two `ForeignPhysicalExpr`s wrapping equal expressions from the same library compare equal. They used to compare by identity. Once requirements cross FFI, that could make planning fail `SanityCheckPlan` when a foreign plan's foreign child already met its partitioning or ordering requirement. Expressions from different libraries always compare unequal, since one library can't safely read another's private data. - New stable types: - `FFI_Distribution` and `FFI_InputDistributionRequirements` in `physical_expr/partitioning.rs`. Co-partitioning crosses as a bool, because `InputDistributionRequirements` can only be built with all children co-partitioned or none. - `FFI_PhysicalSortRequirement` and `FFI_OrderingRequirements` (hard/soft alternatives) in `physical_expr/sort.rs`. - The deprecated `Distribution::HashPartitioned` crosses as `KeyPartitioned`. The optimizer treats the two the same, and this keeps the deprecated variant out of the ABI. - A note in the 56.0.0 upgrade guide covering both layout changes. It is in the 56 guide because this is going into 56.0.0 (see the last section). ## What is the testing strategy for this PR? - `test_ensure_requirements_sees_foreign_plan_requirements` reproduces the bug. It plans `SELECT a % 2, count(*) ... GROUP BY 1` with `target_partitions = 4`, removes the hash repartition, sends the `FinalPartitioned` aggregate across a mock FFI boundary, and runs `EnsureRequirements`. It asserts that a hash repartition is inserted and that `check_plan_tree_sanity` passes. Without the fix it fails: the plan gets `RoundRobinBatch(4)`. - `test_ffi_execution_plan_input_requirements_round_trip`: default and non-default requirements, including co-partitioning, soft ordering and the deprecated per-child accessor, come back unchanged from a foreign plan. - `foreign_requirements_met_by_foreign_child` in the FFI integration tests crosses a real cdylib. The foreign plan requires `KeyPartitioned([a])` over a `Hash([a], 4)` child, or a hard ordering on `a` over a `SortExec` on `a`. `SELECT * FROM t` must plan without adding a repartition or sort. With identity equality it fails `SanityCheckPlan`. - `test_ffi_physical_expr_eq_across_library_images` loads two copies of the test library with `get_module_copy`. The same `Column` from each copy compares unequal, and two from the same copy compare equal. It fails if the library check in `eq` is removed. - Unit round-trip tests for the new FFI types in `partitioning.rs` and `sort.rs`. - End to end: I cherry-picked this onto the `branch-56` commit apache/datafusion-python#1826 builds against. The previously failing `test_spawning_plan_across_three_libraries` now passes, along with the rest of that repo's FFI example suites. That run predates the `eq` callback and the requirement caching. ## Are there any user-facing changes? The `FFI_ExecutionPlan` and `FFI_PhysicalExpr` layouts change, so FFI providers and consumers need to be rebuilt against the same release. Foreign plans now report their real input requirements to optimizer rules. I'll backport this to `branch-56` before 56.0.0 is final (only `56.0.0-rc1` is tagged so far). The datafusion-python upgrade to 56 needs it, and backporting before the release means 56.0.0 and later 56.0.x releases share one layout. The change applies to `branch-56` without conflicts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ackport #27428) (#27541) ## Which issue does this PR close? Backports #27428 (fix for #27427) to `branch-56` for the 56.0.0 release tracked in #24461. ## Rationale for this change The datafusion-python upgrade exposed a planning regression in 56.0.0 RC1. When the query planner and optimizer run on opposite sides of the FFI boundary, the optimizer can miss an aggregate's hash partitioning requirement and produce a plan that fails `SanityCheckPlan`. This blocks apache/datafusion-python#1826. ## What changes are included in this PR? Clean cherry-pick of 701e556. Foreign execution plans expose their input requirements, and physical expressions originating in the same library can be compared for equality. The 56.0.0 upgrade guide describes both ABI changes. ## What is the testing strategy for this PR? Verified on the backport branch: - `cargo clippy --all-targets --all-features -- -D warnings` - `cargo test -p datafusion-ffi -p datafusion-ffi-integration-tests --features integration-tests`, including the `EnsureRequirements` regression and the cross-library plan-requirements and expression-equality tests - `cargo fmt --all` and the repository's Markdown formatting check Local `cargo audit` reports RUSTSEC-2026-0523, RUSTSEC-2026-0524 and RUSTSEC-2026-0538 for the existing `quinn-proto` and `quinn-udp` versions. The unmodified RC1 lockfile reproduces all three findings. This backport preserves the dependency versions. ## Are there any user-facing changes? Yes. Foreign plans preserve their input requirements. The layouts of `FFI_ExecutionPlan` and `FFI_PhysicalExpr` change. FFI providers and consumers built against RC1 must be rebuilt together for RC2. Including the fix before 56.0.0 is final keeps one ABI for the published 56.x series. Co-authored-by: Andy Grove <agrove@apache.org>
Which issue does this PR close?
N/A.
Rationale for this change
A new DataFusion version is being released: apache/datafusion#24461
What changes are included in this PR?
There are new types like
Show(which replacedDescribeTable),AsOfJoin, andMaterializedCte, as well as some other minor changes. I will add them as comments.Are there any user-facing changes?
New DF version. Other than that the main breaking change is that
DescribeTablehas been removed. I added a note to the upgrade guide.