This document describes how to sync datafusion-python to a new version of the
upstream apache/datafusion Rust crates. This is a recurring task: between
official releases the main branch tracks DataFusion via crates.io or GitHub
dependencies, and we periodically bump those dependencies to pick up new
features and bug fixes.
The work is broken into three sequential PRs rather than landing as one large change. Splitting reviews along these lines keeps each PR focused, makes breakage easier to bisect, and lets reviewers concentrate on one concern at a time.
Goal: update the upstream datafusion crate version and make the project
build, test, and lint cleanly against it.
-
In the root
Cargo.toml, update:[workspace.package].versionto the new major (thedatafusion-pythonmajor tracks the upstreamdatafusionmajor, so a 53→54 bump moves this from53.0.0to54.0.0), and- every
datafusion/datafusion-*entry in[workspace.dependencies]to the same new major.
Per-crate manifests under
crates/inherit these pins viaworkspace = trueand need no edit. -
Update
Cargo.lockfor the datafusion family only — leave unrelated transitives at their current pins so PR 2 can address them deliberately. List everydatafusion-*workspace dependency with-p:cargo update \ -p datafusion \ -p datafusion-substrait \ -p datafusion-proto \ -p datafusion-ffi \ -p datafusion-catalog \ -p datafusion-common \ -p datafusion-functions-aggregate \ -p datafusion-functions-window \ -p datafusion-expr
Or pin exact versions with
--precise, one crate at a time:cargo update -p datafusion --precise 54.0.0 # repeat for each datafusion-* siblingA bare
cargo updatewould refresh every transitive crate and blur the diff between PR 1 and PR 2. -
Run the standard build and test commands and address compilation errors, API renames, signature changes, and behavior changes:
cargo buildcargo testpytestpre-commit run --all-files
-
Fix only what's needed to restore green CI. Resist the urge to bundle unrelated cleanups — those belong in their own PR.
-
If a breaking change in upstream requires a user-facing API change in
datafusion-python, add theapi changelabel and document the change in the PR description so it surfaces in the changelog.
Reference PRs: #1311 (DF51), #1337 (DF52).
Goal: after the upstream bump, the dependency tree may have multiple
versions of the same transitive crate (for example, two arrow versions, two
object_store versions). Reconcile these so we ship a single coherent set.
- Inspect the lockfile for duplicates:
cargo tree --duplicates
- For each duplicate that matters (Arrow,
object_store,parquet,tokio,arrow-flight, etc.), update our direct dependency declarations inCargo.tomlto versions compatible with what upstream DataFusion now pulls in. The goal is one version of each ecosystem-critical crate. - Re-run
cargo updateand re-run the full test matrix. Some duplicates are benign (small leaf crates with no FFI surface) and can be left alone if reconciliation would force a much larger change. Use judgment. - If consolidating forces a behavioral change visible to users (for example,
a newer
pyarrow-compatible Arrow version), call it out in the PR description.
Keeping this work separate from PR 1 means PR 1 stays a "make it compile" review and PR 2 stays a "tidy the dependency graph" review.
Goal: with the upstream version locked in, identify new APIs that landed upstream and decide whether to expose them, and update agent-facing documentation so it still matches the surface we ship.
- Run the
check-upstreamskill (.ai/skills/check-upstream/SKILL.md) to diff the upstream Rust API against what's exposed inpython/datafusion/. The skill covers scalar/aggregate/window/table functions,DataFramemethods,SessionContextmethods, and FFI types. Invoke it from the assistant with/check-upstream(optionally scoped to one area, e.g./check-upstream scalar functions). - For each gap, decide whether to:
- Expose it now (small, obvious additions can land in this PR).
- File a tracking issue (anything non-trivial — separate PR per feature keeps reviews focused).
- Skip it (internal-only or already covered by an existing API; record the decision in the "Evaluated and not requiring exposure" sections of the skill so future runs don't re-flag it).
- (Optional) Run the
make-pythonicskill (.ai/skills/make-pythonic/SKILL.md) over any newly exposed APIs to align signatures with the project's Pythonic style (accepting plain strings for column names, raw Python values where auto-wrapping applies, etc.). Invoke it from the assistant with/make-pythonic. Running this before the audit step means examples inSKILL.mdget updated to the final signature in one pass instead of churning twice. Larger reshapes still belong in their own PR. - Run the
audit-skill-mdskill (.ai/skills/audit-skill-md/SKILL.md) to cross-reference the user-facing skill atskills/datafusion_python/SKILL.mdagainst the current public API. The skill flags stale function names, missing newly exposed APIs, examples that drifted from idiomatic style, and missing version notes. Invoke it from the assistant with/audit-skill-md(optionally scoped, e.g./audit-skill-md dataframe). Apply the resulting edits toSKILL.mdand to the relevant RST pages underdocs/source/user-guide/common-operations/. - If new aggregate or window functions were exposed in step 2, also update:
docs/source/user-guide/common-operations/aggregations.rstdocs/source/user-guide/common-operations/windows.rst
- Bisectable. If a regression appears,
git bisectlands on the responsible PR (compile fix, dependency consolidation, or API addition) rather than a single mega-commit. - Reviewable. Each PR has a single concern. Reviewers reading PR 1 don't need to also reason about whether new APIs are well-named.
- Skippable. Some upstream syncs are pure version bumps with no new APIs worth exposing. PR 3 can be empty or merged as a no-op if the audit comes back clean.
README.md— the broader release process (this sync work feeds into the next official release)..ai/skills/check-upstream/SKILL.md— API coverage audit.skills/datafusion_python/SKILL.md— user-facing agent guide kept in sync via PR 3.