Repository navigation
[DISCUSSION] 2026 Q4 Roadmap #26684
Description
- part of [Meta-EPIC] Organizing DataFusion's Growth #21036
- Follow on to [DISCUSSION] 2026 Q3-Q4 Roadmap Discussion #22882
As I have said before, it is still mind boggling 🤯 to me that DataFusion has gotten so far without central planning authority, but the decentralized model has served us (very) well and I don't think we need to change. However, the rise of AI generated code, especially in the last few months, has been challenging this model as it is so easy to generate changes, I think the collective attention of the community has been overwhelmed and we are thrashing. See for example these discussions
- [DISCUSSION] Rethinking Code Review and Testing in the Agent Era #23839 from @jayzhan211
- Policy on uptick of LLM PRs from new contributors #24413 from @Jefffrey
- Update contributor guidelines regarding AI spam, reviews, code #24510
- ci: Enforce PR limit to 3 for non-committers #25973 from @2010YOUY01
What I suggest is that a few maintainers agree to help one or more projects along and then prioritize reviews of related PRs.
As always, I expect DataFusion priorities to be driven by what the maintainers want to help drive to completion. However, I also personally think the community would benefit from some agreement on higher level "Light House" areas to focus our collective attention on before we wade into the sea of PRs. Since I am just one voice, I don't get to choose what everyone else works on, but I here are some of my suggestions to get the conversation flowing.
Tier 1: Correctness
What: Ensure answers are always correct
Goal: Have DataFusion be the standard to compare against
Tracking Epic:
Tier 1: Performance (TBD)
I think it would be great to agree on or two areas to really focus and improve performance. My suggestion is either Parquet IO Patterns and Join benchmark performance, but I think there are other compelling areas to
Some options:
Performance Option: Parquet IO patterns
What: Make the I/O pattern of Parquet scans deliberate and configurable (one request per row group, bounded prefetch, or batch-granular fetch) instead of a hard coded trade-off between latency and memory
Goal: lower time to first batch and peak memory per partition, overlap I/O with decode, and let the scan adapt to the storage system (object store vs local SSD)
Recent Epics:
Performance Option: Join Benchmark Performance
What: Improve performance in Join heavy benchmarks
Goal: ensure we keep DataFusion at the top of benchmarks (especially join heavy ones),
Recent Epics:
Performance Option: Filter Pushdown
What: Enable Parquet filter pushdown (late materialization) by default and finish the logical filter / leaf expression pushdown work
Goal: evaluate filters as early as possible (during the scan where it helps) without performance regressions
Recent Epics:
- [EPIC] Adaptive predicate evaluation #22883
- [EPIC] Fix performance regressions when enabling parquet filter pushdown (late materialization) #20324
- [EPIC] Leaf expression pushdown and filter pushdown: bugs, decisions and tests #25459
- [Epic] Dynamic filtering related items #15512
Performance Option: Nested and Semi-structured Data
What: Improve struct field access, nested schema pruning and Variant support in the Parquet scan
Goal: make queries over nested columns as efficient as over flat ones
Recent Epics:
- [EPIC] Improvements to struct access in Parquet #24119
- [EPIC] first class support for struct field / Variant access in Parquet #21308
- [EPIC] Variant on DataFusion #26517
Other potential projects we might want to rally around:
- Spilling Joins: [EPIC] Spilling Hash Join — run any join in a bounded memory budget #24768
- blocked / chunked memory in hash aggregation: [EPIC] Use blocked / chunked memory management in hash aggregation #24704
- Optimize range partitioning for joins and distributed pipelines: [EPIC] Optimize Range Partitioning for Joins and Distributed Pipelines #25421
- Subplan materialization: [EPIC] Subplan Materalization #22676
- Adaptive predicate evaluation: [EPIC] Adaptive predicate evaluation #22883
Things I personally plan to deemphasize
- Seemingly random improvements to random parts of the code, especially if it isn't from a DataFusion user or from a Prolific Author
Activity
2010YOUY01 commentedon Oct 24, 2026
My personal focus is:
Feature Improvements
Improve join ordering as the goal, and clear the blockers and improve maintainability in the optimizer/operators/statistics framework along the way.
I'm still investigating and prototyping locally, and hope to share some results soon.
Community Building
Figure out better ways to deal with AI spam.
One thing I feel I'm doing less of now is reviewing PRs from new contributors. I'm aware this is not ideal — we should help more contributors grow into long-term maintainers for the health of the project.
The challenge is that it's hard to judge from their initial PRs whether they're responsible contributors or irresponsible prolific PR authors. Reviewing PRs from the latter is definitely a waste of time. I still have no clue how to distinguish them, especially as coding agents are getting better. Contributors from known DataFusion users seem to be the only reliable signal I'm aware of. e.g., contributors from major df users like Datadog are always great to collaborate with.
I think we should be more aggressive about closing PRs from irresponsible prolific authors. Reviewing their PRs, and encouraging them to submit more PRs might further waste reviewing bandwidth.
jayzhan211 commentedon Oct 24, 2026
Spilling HJ #24768
Join Ordering +1
Physical Optimizer Revisit #26481
Aggregation (anything related; currently focused on #24704)
Refactor / Simplification (structural, not cosmetic; feel free to ping me)
Leverage AI #23839
kumarUjjawal commentedon Oct 24, 2026
As I have been doing for the past several weeks. I will continue to spend almost all of my bandwidth on PR reviews and making sure CI is working fine for all the contributors. If anyone has prs which require review, please don't hesitate to ping me as I will be available for reviews.
alamb commentedon Oct 24, 2026
Thank you. I have been trying to clear out the merge queue this morning. I am also planning to ask much more from contributors before merging their PRs (like they need to address all comments from approvers). I hvae also been aggressively marking PRs as draft (so I can filter them out of lists) when the authors need to respond to feedback
kumarUjjawal commentedon Oct 24, 2026
Another thing that came to my mind was that currently we let PR sit for 60 days before marking it stale and eventually closing it after 7 days. Would it make sense to make the pr as draft automatically after 14 to 21 days of inactivity and the author can ping reviewers if there's any activity.
alamb commentedon Oct 24, 2026
I think we should try it as the cost is low (the author can reopen / comment on it)
jayzhan211 commentedon Oct 24, 2026
Auto-drafting sounds like a good idea. One suggestion: could we limit it to PRs that are waiting on the author, i.e. idle for 14+ days and with a merge conflict, failing CI, or a "changes requested" review newer than the last push?
On today's open PRs, a plain 14-day rule would also move 17 PRs that are waiting on a reviewer (7 of them never reviewed) and 4 approved ones out of the
draft:falseview. The scoped rule would still catch all 66 author-side ones.apache/arrow already converts PRs to draft from a workflow with the default token (
.github/workflows/dev_pr.yml), so this might be a small job. @kumarUjjawal what do you think about using that condition in the workflow?kumarUjjawal commentedon Oct 24, 2026
I did some digging and I agree this would be a good idea, especially regarding the pr not reviewed yet. Thanks for the suggestion.
Omega359 commentedon Oct 24, 2026
I hope to finally finish off the migration from native to sql benchmarking:
bench.shto use the newbenchmark_runner#24311dfbenchrunner once benchmarks are migrated to the new SQL benchmark framework #24312I intend to do some more reviews of PRs though I will likely limit what I review to PRs that interest or impact me personally (sql benchmarks, udfs and date/time work). I've done a few reviews of 'random' PRs and only about half actually seem to respond. I've had the idea of first posting a random question to those types of PRs that may interest me to review and if I actually get a response then maybe I'll actually take the effort to review the change.
ratmice commentedon Oct 24, 2026
One thing I wonder is whether there is automatic labeling for the crates that a pr touches,
I've been somewhat trying to follow and add my two cents onto PRs for the datafusion-cli crate, but with no label (#24021)
I'm somewhat wondering if there is a bot/method to automatically apply labels based on whether they touch (or even just contain mostly changes to a particular crate).
Stuff like datafusion-cli doesn't really seem to be a priority, but it'd be good to be able to search more easily for PRs affecting specific crates.
alamb commentedon Oct 25, 2026
yes there are already labels and automatic labeling. We could probably add one for datafusion-cli more easily
ci: draft inactive PRs awaiting author action (apache#26715)