You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
In the upstream repository we have functions that have been built for Spark compatibility. If we make these functions available in datafusion-python then we will both expand the usability of our project and also make it easier for new users to convert their projects from spark -> datafusion-python workflows.
Describe the solution you'd like
Evaluate all of the features that are in the upstream repository and create a list of what features we wish to expose. I have not spent a lot of time reviewing the upstream spark code so whoever takes ownership of this issue should spend time understanding all that is available and in progress.
Create a method to register all the spark functions with the session context.
Expose all of the functions via the DataFrame API.
Add test coverage
Add documentation for the user site about how to enable the spark functions and a description of why they are not enabled by default.
Describe alternatives you've considered
We could release these as a separate Python package.
Additional context
Some of these functions replace the default functions and some are unique. For users of the DataFrame API this is likely not a problem because we can put them under a spark module, such as datafusion.spark.functions or datafusion.functions.spark. The big difference will be for those who use the SQL approach. We will probably want a function that updates the session context to register the spark functions on top of the default functions.
Big fan of this for making datafusion-python more accessible to pyspark users. Especially if this means we potentially push for mllib coverage from spark. I often wish for more operations across arrays etc element wise diff of array/list, lp norms, ...
I don't think I have bandwidth in the near term to push on it heavily though
After some back and forth with an agent, we came up with this plan. This does not need to be the final plan, but it looks good to me as a starting point.
Plan: Expose datafusion-spark Functions to Python
Context
The datafusion-spark Rust crate (in apache/datafusion at datafusion/spark/) provides ~74 scalar functions and 4 aggregate functions with Apache Spark-compatible semantics. We want to expose these in datafusion-python so users can opt into Spark-compatible behavior.
The core challenge: ~14-15 spark functions share names with existing DataFusion built-ins but have different behavioral semantics. This affects both SQL (where functions are resolved by name from a registry) and the DataFrame API (where functions are bound at expression construction time).
Name Collisions Inventory
High-Impact Collisions (different results for same input)
Function
Spark Behavior
DataFusion Behavior
concat
Returns NULL if any input is NULL
Skips NULLs
round
HALF_UP rounding (2.5 → 3)
Banker's rounding (2.5 → 2)
date_part
1-indexed day-of-week (Sun=1)
0-indexed day-of-week
date_trunc
Converts to session TZ, always returns microseconds
Preserves original precision, no TZ conversion
substring/substr
Negative start counts from end
Different negative-index handling
trunc
Date truncation function
Math truncation (toward zero) — completely different
abs
Returns min value on signed overflow
Errors on overflow
array_contains
Three-valued NULL logic (NULL if element missing but array has NULLs)
Returns false
Moderate-Impact Collisions
Function
Difference
avg (aggregate)
Internal count uses i64 vs u64
ascii
Spark accepts numeric types via implicit cast
factorial
Spark restricts to Int32, returns NULL for out-of-range
For SQL queries, function resolution happens through the SessionContext's function registry. The datafusion-spark crate provides register_all(&mut FunctionRegistry) which registers all spark functions, silently overwriting any existing function with the same name.
Add an explicit method on SessionContext, following the existing enable_url_table() pattern, plus a selective registration method for users who want fine-grained control:
ctx=SessionContext()
# Option 1: Register ALL spark functions (overwrites collisions)ctx.enable_spark_functions()
# Option 2: Register a single spark function by namectx.register_spark_function("round")
# SQL now uses Spark semantics for registered colliding namesresult=ctx.sql("SELECT concat('a', NULL, 'b')").collect()
# Returns NULL (Spark) instead of 'ab' (DataFusion)# Original DataFusion version still accessible via df_ prefixresult=ctx.sql("SELECT df_concat('a', NULL, 'b')").collect()
# Returns 'ab' (DataFusion)
Implementation:
Rust (context.rs): Add enable_spark_functions(&self) that registers all spark functions, and register_spark_function(&self, name: &str) for selective registration. Both preserve originals under df_ prefix before overwriting.
Python (context.py): Thin wrappers calling the Rust methods.
Preserving originals for SQL: Before overwriting a colliding function, re-register the original DataFusion function under a df_ prefix (e.g., df_concat, df_round). Users who need both semantics in one SQL query can use the prefixed name:
Selective registration is useful for users who only want specific spark behaviors (e.g., Spark's round semantics) without changing all colliding functions. The selective method also registers non-colliding spark-only functions (e.g., ctx.register_spark_function("crc32")).
Part 2: DataFrame API Integration — Approaches
For the DataFrame API, the situation is fundamentally different: Python functions like f.round(col("x")) bind directly to a function implementation at expression construction time (not via name lookup). So even if spark functions are registered on the context for SQL, datafusion.functions.round still uses the DataFusion implementation.
Approach A: Separate datafusion.spark.functions Module (Recommended)
All spark functions live in a dedicated namespace:
fromdatafusionimportfunctionsasffromdatafusion.sparkimportfunctionsassf# DataFusion semanticsdf.select(f.round(col("x"), lit(2))) # banker's rounding# Spark semantics df.select(sf.round(col("x"), lit(2))) # HALF_UP rounding# Spark-only function (no collision)df.select(sf.crc32(col("data")))
Pros:
Zero breaking changes — datafusion.functions is untouched
Explicit about which semantics you're using — no ambiguity
Clean IDE autocompletion: sf. shows all spark functions, f. shows all DF functions
Familiar to PySpark users (pyspark.sql.functions → datafusion.spark.functions)
Future-proof: same pattern works for datafusion.duckdb.functions etc.
All spark functions (colliding and non-colliding) live together consistently
Matches how the Rust crate is organized (datafusion-spark is its own crate)
Cons:
Users who want spark-only functions (no collision) must import from the spark module
Puts "spark" in the module path — see naming discussion below
The ~60 non-colliding spark functions are less discoverable than if they were in datafusion.functions
Approach B: Merge Everything into datafusion.functions
Add all spark functions into the existing module. For collisions, use prefixed names (spark_round, spark_concat). Non-colliding functions are added without prefix.
Inconsistent naming: spark-only functions have no prefix, colliding ones do. Is f.crc32 a "spark function"? Is f.round not?
Pollutes autocomplete with spark_* variants interleaved with regular functions
Already a 6231-line file — grows further
spark_ prefix is ugly and doesn't match how the functions are named in Spark itself
No clean way to "import all spark functions" — they're mixed in
Approach C: Config-Based Switching
A global/session config changes what f.round(...) resolves to:
ctx=SessionContext()
ctx.set_function_variant("spark")
df.select(f.round(col("x"), lit(2))) # now uses Spark semantics
Pros:
Minimal API change for users migrating from PySpark wholesale
Same code works for both variants
Cons:
Dangerous implicit behavior: same code produces different results depending on hidden state
Makes code reviews/debugging harder — f.round(...) has no visible indication of which implementation
Thread safety concerns with global state
Mixed usage (needing both semantics in one query) is awkward
Violates "explicit is better than implicit" (Python Zen)
IDE/type checkers can't distinguish variants
Naming Concern: "spark" in the codebase
The user raised whether putting spark in module paths sends the wrong message.
Considerations:
The upstream Rust crate is officially named datafusion-spark in the Apache DataFusion project — this isn't our invention
Users looking for Spark compatibility will search for "spark" — discoverability matters
Alternative names (extended, compat, extras) are vague and don't convey what the functions actually do
The substrait module already establishes a precedent for naming submodules after the external standard they integrate with
Recommendation: Use spark in the module name. It's accurate, discoverable, and matches the upstream crate name.
Recommended Approach Summary
SQL:ctx.enable_spark_functions() — all-or-nothing, explicit opt-in, preserves originals under df_ prefix.
DataFrame API: Approach A — datafusion.spark.functions as a separate module. This is the cleanest separation and matches Python idioms.
Implementation Plan
Step 1: Add datafusion-spark as a default-on dependency
Files:
/Cargo.toml (workspace): Add datafusion-spark = { version = "53" } to [workspace.dependencies]
/crates/core/Cargo.toml: Add datafusion-spark = { workspace = true, optional = true } and feature spark = ["dep:datafusion-spark"]. Add "spark" to the default feature list: default = ["mimalloc", "spark"]
/pyproject.toml: Add "spark" to features = ["substrait"] → features = ["substrait", "spark"]
The spark feature is enabled by default, matching the substrait precedent. Users building custom wheels can opt out with --no-default-features if needed.
Step 2: Rust — Add enable_spark_functions() and register_spark_function() to PySessionContext
File:crates/core/src/context.rs
Add methods (gated with #[cfg(feature = "spark")]):
#[cfg(feature = "spark")]constSPARK_COLLISIONS:&[&str] = &["abs","ascii","avg","concat","date_part","date_trunc","factorial","length","round","soundex","substring","trunc","array_contains","array_repeat","array",];#[cfg(feature = "spark")]fnpreserve_original_if_collision(&self,name:&str) -> PyDataFusionResult<()>{ifSPARK_COLLISIONS.contains(&name){ifletOk(original) = self.ctx.udf(name){let renamed = original.as_ref().clone().with_name(format!("df_{name}"));self.ctx.register_udf(renamed);}}Ok(())}#[cfg(feature = "spark")]pubfnenable_spark_functions(&self) -> PyDataFusionResult<()>{// Save originals under df_ prefix for all colliding functionsfor name inSPARK_COLLISIONS{self.preserve_original_if_collision(name)?;}// Register all spark functionsfor udf in datafusion_spark::all_default_scalar_functions(){self.ctx.register_udf((*udf).clone());}for udaf in datafusion_spark::all_default_aggregate_functions(){self.ctx.register_udaf((*udaf).clone());}Ok(())}#[cfg(feature = "spark")]pubfnregister_spark_function(&self,name:&str) -> PyDataFusionResult<()>{// Look up the spark function by name across all function typeslet all_scalars = datafusion_spark::all_default_scalar_functions();ifletSome(udf) = all_scalars.iter().find(|f| f.name() == name){self.preserve_original_if_collision(name)?;self.ctx.register_udf((**udf).clone());returnOk(());}let all_aggs = datafusion_spark::all_default_aggregate_functions();ifletSome(udaf) = all_aggs.iter().find(|f| f.name() == name){self.preserve_original_if_collision(name)?;self.ctx.register_udaf((**udaf).clone());returnOk(());}Err(PyValueError::new_err(format!("No spark function named '{name}' found")).into())}
defenable_spark_functions(self) ->None:
"""Register all Spark-compatible functions on this context. Colliding functions are overwritten with Spark semantics. Original DataFusion versions remain accessible with a ``df_`` prefix (e.g., ``df_round``, ``df_concat``). """self.ctx.enable_spark_functions()
defregister_spark_function(self, name: str) ->None:
"""Register a single Spark-compatible function by name. If the function collides with an existing DataFusion built-in, the original is preserved with a ``df_`` prefix. Args: name: The name of the spark function to register (e.g., "round", "crc32"). Raises: ValueError: If no spark function with the given name exists. """self.ctx.register_spark_function(name)
Step 3: Rust — Create spark_functions.rs module
File:crates/core/src/spark_functions.rs (new)
Define #[pyfunction] wrappers for each spark function. For each of the ~74 scalar functions:
#[pyfunction]fnround(value:PyExpr,decimal_places:Option<PyExpr>) -> PyExpr{// Use the spark UDF directly to create the expressionlet spark_round_udf = datafusion_spark::spark_round();// or equivalent accessorExpr::ScalarFunction(ScalarFunction::new_udf(spark_round_udf, args)).into()}
For the 4 aggregate functions, follow the add_builder_fns_to_aggregate pattern from functions.rs.
Define pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> that registers all functions.
Note: The exact API for constructing expressions from datafusion-spark UDFs needs verification against the crate's public API. It may provide expr_fn::round(...) convenience functions or require ScalarFunction::new_udf().
Functions in spark/functions.py raise a clear error if the feature isn't compiled.
Step 7: Testing
File:python/tests/test_spark_functions.py (new)
Test all spark functions for correctness
For colliding functions, add comparison tests demonstrating the behavioral difference:
deftest_round_spark_vs_default():
# DataFusion uses banker's roundingassertf.round(lit(2.5)).collect() ==2# round to even# Spark uses HALF_UPassertsf.round(lit(2.5)).collect() ==3# round away from zero
Test enable_spark_functions() works for SQL queries
Test register_spark_function("round") registers only that one function
Test that register_spark_function("nonexistent") raises ValueError
Test that df_ prefixed originals are accessible after enabling spark functions
Step 8: Documentation
Add docs page at docs/source/user-guide/spark-functions.rst explaining the module
Document all collisions with behavioral differences
Add migration guide for PySpark users
Verification Plan
Build:maturin develop compiles successfully (spark enabled by default)
Import test:from datafusion.spark import functions as sf works
SQL test (enable all): After ctx.enable_spark_functions(), SELECT round(2.5) returns 3 (Spark HALF_UP)
SQL test (selective): After ctx.register_spark_function("round"), SELECT round(2.5) returns 3 but SELECT concat('a', NULL, 'b') still returns 'ab' (DataFusion)
SQL test (selective error):ctx.register_spark_function("nonexistent") raises ValueError
How about standardizing on spark API as the default, there is really no point in the long term to have two API ?
I don't think we'd want to standardize on spark as default, especially since some functions return differently. Then anyone who is expected PostgreSQL-like queries to work would fail / return unexpected results. Do you disagree?
It feels to me like enabling spark functions in the SQL syntax needs to be an opt-in feature.
I was referring more to the syntax, since it makes it easier for both humans and agents to write familiar code. To be clear, I don’t think matching Spark’s output is a good idea, especially since even Spark is moving toward ANSI compliance by default.
I was referring more to the syntax, since it makes it easier for both humans and agents to write familiar code. To be clear, I don’t think matching Spark’s output is a good idea, especially since even Spark is moving toward ANSI compliance by default.
I'm sorry, but I don't understand what you're specifically recommending.
Activity
ntjohnson1 commentedon Apr 8, 2026
Big fan of this for making datafusion-python more accessible to pyspark users. Especially if this means we potentially push for mllib coverage from spark. I often wish for more operations across arrays etc element wise diff of array/list, lp norms, ...
I don't think I have bandwidth in the near term to push on it heavily though
timsaucer commentedon Apr 9, 2026
After some back and forth with an agent, we came up with this plan. This does not need to be the final plan, but it looks good to me as a starting point.
Plan: Expose
datafusion-sparkFunctions to PythonContext
The
datafusion-sparkRust crate (inapache/datafusionatdatafusion/spark/) provides ~74 scalar functions and 4 aggregate functions with Apache Spark-compatible semantics. We want to expose these indatafusion-pythonso users can opt into Spark-compatible behavior.The core challenge: ~14-15 spark functions share names with existing DataFusion built-ins but have different behavioral semantics. This affects both SQL (where functions are resolved by name from a registry) and the DataFrame API (where functions are bound at expression construction time).
Name Collisions Inventory
High-Impact Collisions (different results for same input)
concatrounddate_partdate_truncsubstring/substrtruncabsarray_containsModerate-Impact Collisions
avg(aggregate)asciifactoriallength/char_lengtharray_repeatsoundexarrayNon-Colliding Functions (~60, safe to add)
add_months,base64,bin,bit_count,bit_get,bitmap_*,bitwise_not,char,collect_list,collect_set,crc32,csc,date_add,date_diff,date_sub,elt,expm1,format_string/printf,from_utc_timestamp,hex,hour,if,ilike,json_tuple,last_day,like,luhn_check,make_dt_interval,make_interval,map_from_entries,minute,mod,negative,next_day,parse_url,pmod,rint,sec,second,sha1/sha,sha2,shiftleft,shiftright,shiftrightunsigned,shuffle,size,slice,space,spark_cast,str_to_map,time_trunc,to_utc_timestamp,try_parse_url,try_sum,try_url_decode,unbase64,unhex,unix_date,unix_micros,unix_millis,unix_seconds,url_decode,url_encode,width_bucketPart 1: SQL Integration
For SQL queries, function resolution happens through the SessionContext's function registry. The
datafusion-sparkcrate providesregister_all(&mut FunctionRegistry)which registers all spark functions, silently overwriting any existing function with the same name.Approach:
ctx.enable_spark_functions()+ selectivectx.register_spark_function(name)Add an explicit method on
SessionContext, following the existingenable_url_table()pattern, plus a selective registration method for users who want fine-grained control:Implementation:
context.rs): Addenable_spark_functions(&self)that registers all spark functions, andregister_spark_function(&self, name: &str)for selective registration. Both preserve originals underdf_prefix before overwriting.context.py): Thin wrappers calling the Rust methods.Preserving originals for SQL: Before overwriting a colliding function, re-register the original DataFusion function under a
df_prefix (e.g.,df_concat,df_round). Users who need both semantics in one SQL query can use the prefixed name:Selective registration is useful for users who only want specific spark behaviors (e.g., Spark's
roundsemantics) without changing all colliding functions. The selective method also registers non-colliding spark-only functions (e.g.,ctx.register_spark_function("crc32")).Part 2: DataFrame API Integration — Approaches
For the DataFrame API, the situation is fundamentally different: Python functions like
f.round(col("x"))bind directly to a function implementation at expression construction time (not via name lookup). So even if spark functions are registered on the context for SQL,datafusion.functions.roundstill uses the DataFusion implementation.Approach A: Separate
datafusion.spark.functionsModule (Recommended)All spark functions live in a dedicated namespace:
Pros:
datafusion.functionsis untouchedsf.shows all spark functions,f.shows all DF functionspyspark.sql.functions→datafusion.spark.functions)datafusion.duckdb.functionsetc.datafusion-sparkis its own crate)Cons:
datafusion.functionsApproach B: Merge Everything into
datafusion.functionsAdd all spark functions into the existing module. For collisions, use prefixed names (
spark_round,spark_concat). Non-colliding functions are added without prefix.Pros:
f.Cons:
f.crc32a "spark function"? Isf.roundnot?spark_*variants interleaved with regular functionsspark_prefix is ugly and doesn't match how the functions are named in Spark itselfApproach C: Config-Based Switching
A global/session config changes what
f.round(...)resolves to:Pros:
Cons:
f.round(...)has no visible indication of which implementationNaming Concern: "spark" in the codebase
The user raised whether putting
sparkin module paths sends the wrong message.Considerations:
datafusion-sparkin the Apache DataFusion project — this isn't our inventionextended,compat,extras) are vague and don't convey what the functions actually dosubstraitmodule already establishes a precedent for naming submodules after the external standard they integrate withRecommendation: Use
sparkin the module name. It's accurate, discoverable, and matches the upstream crate name.Recommended Approach Summary
SQL:
ctx.enable_spark_functions()— all-or-nothing, explicit opt-in, preserves originals underdf_prefix.DataFrame API: Approach A —
datafusion.spark.functionsas a separate module. This is the cleanest separation and matches Python idioms.Implementation Plan
Step 1: Add
datafusion-sparkas a default-on dependencyFiles:
/Cargo.toml(workspace): Adddatafusion-spark = { version = "53" }to[workspace.dependencies]/crates/core/Cargo.toml: Adddatafusion-spark = { workspace = true, optional = true }and featurespark = ["dep:datafusion-spark"]. Add"spark"to thedefaultfeature list:default = ["mimalloc", "spark"]/pyproject.toml: Add"spark"tofeatures = ["substrait"]→features = ["substrait", "spark"]The
sparkfeature is enabled by default, matching thesubstraitprecedent. Users building custom wheels can opt out with--no-default-featuresif needed.Step 2: Rust — Add
enable_spark_functions()andregister_spark_function()toPySessionContextFile:
crates/core/src/context.rsAdd methods (gated with
#[cfg(feature = "spark")]):File:
python/datafusion/context.py— Add Python wrappers:Step 3: Rust — Create
spark_functions.rsmoduleFile:
crates/core/src/spark_functions.rs(new)Define
#[pyfunction]wrappers for each spark function. For each of the ~74 scalar functions:For the 4 aggregate functions, follow the
add_builder_fns_to_aggregatepattern fromfunctions.rs.Define
pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()>that registers all functions.Note: The exact API for constructing expressions from
datafusion-sparkUDFs needs verification against the crate's public API. It may provideexpr_fn::round(...)convenience functions or requireScalarFunction::new_udf().Step 4: Rust — Register spark_functions submodule
File:
crates/core/src/lib.rsStep 5: Python — Create
datafusion.sparkpackageFiles:
python/datafusion/spark/__init__.py— Minimal, exportsfunctionspython/datafusion/spark/functions.py— Python wrappers for all spark functionsPattern for each function:
Each colliding function's docstring should note the behavioral difference from the DataFusion built-in.
File:
python/datafusion/__init__.py— Addsparkto imports and__all__.Step 6: Handle feature flag gracefully in Python
In
python/datafusion/spark/__init__.py:Functions in
spark/functions.pyraise a clear error if the feature isn't compiled.Step 7: Testing
File:
python/tests/test_spark_functions.py(new)enable_spark_functions()works for SQL queriesregister_spark_function("round")registers only that one functionregister_spark_function("nonexistent")raises ValueErrordf_prefixed originals are accessible after enabling spark functionsStep 8: Documentation
docs/source/user-guide/spark-functions.rstexplaining the moduleVerification Plan
maturin developcompiles successfully (spark enabled by default)from datafusion.spark import functions as sfworksctx.enable_spark_functions(),SELECT round(2.5)returns 3 (Spark HALF_UP)ctx.register_spark_function("round"),SELECT round(2.5)returns 3 butSELECT concat('a', NULL, 'b')still returns'ab'(DataFusion)ctx.register_spark_function("nonexistent")raises ValueErrorsf.round(lit(2.5))returns 3,f.round(lit(2.5))returns 2SELECT df_round(2.5)returns 2sf.crc32(lit(b"hello"))workssparkfeature,from datafusion.spark import functionsraises clear ImportErrorpython/tests/Key Files to Modify/Create
/Cargo.tomldatafusion-sparkto workspace deps/crates/core/Cargo.tomlsparkfeature/pyproject.toml"spark"to maturin features/crates/core/src/context.rsenable_spark_functions()/crates/core/src/spark_functions.rs/crates/core/src/lib.rs/python/datafusion/spark/__init__.py/python/datafusion/spark/functions.py/python/datafusion/__init__.py/python/datafusion/context.pyenable_spark_functions()wrapper/python/tests/test_spark_functions.pydjouallah commentedon Apr 11, 2026
How about standardizing on spark API as the default, there is really no point in the long term to have two API ?
timsaucer commentedon Apr 11, 2026
I don't think we'd want to standardize on spark as default, especially since some functions return differently. Then anyone who is expected PostgreSQL-like queries to work would fail / return unexpected results. Do you disagree?
It feels to me like enabling spark functions in the SQL syntax needs to be an opt-in feature.
djouallah commentedon Apr 12, 2026
I was referring more to the syntax, since it makes it easier for both humans and agents to write familiar code. To be clear, I don’t think matching Spark’s output is a good idea, especially since even Spark is moving toward ANSI compliance by default.
timsaucer commentedon Apr 12, 2026
I'm sorry, but I don't understand what you're specifically recommending.
djouallah commentedon Apr 12, 2026
it will be nice maybe to use the same spark api syntax, even if the output does not 100 % match the expected behavior of Spark
feat: expose spark-compatible functions (#1564)