Skip to content

Expose Spark functions #1482

Closed
Closed

Description

@timsaucer

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

In the upstream repository we have functions that have been built for Spark compatibility. If we make these functions available in datafusion-python then we will both expand the usability of our project and also make it easier for new users to convert their projects from spark -> datafusion-python workflows.

Describe the solution you'd like

  • Evaluate all of the features that are in the upstream repository and create a list of what features we wish to expose. I have not spent a lot of time reviewing the upstream spark code so whoever takes ownership of this issue should spend time understanding all that is available and in progress.
  • Create a method to register all the spark functions with the session context.
  • Expose all of the functions via the DataFrame API.
  • Add test coverage
  • Add documentation for the user site about how to enable the spark functions and a description of why they are not enabled by default.

Describe alternatives you've considered

We could release these as a separate Python package.

Additional context

Some of these functions replace the default functions and some are unique. For users of the DataFrame API this is likely not a problem because we can put them under a spark module, such as datafusion.spark.functions or datafusion.functions.spark. The big difference will be for those who use the SQL approach. We will probably want a function that updates the session context to register the spark functions on top of the default functions.

Activity

  1. ntjohnson1 commented on Apr 8, 2026

    @ntjohnson1
    Contributor

    Big fan of this for making datafusion-python more accessible to pyspark users. Especially if this means we potentially push for mllib coverage from spark. I often wish for more operations across arrays etc element wise diff of array/list, lp norms, ...

    I don't think I have bandwidth in the near term to push on it heavily though

  2. timsaucer commented on Apr 9, 2026

    @timsaucer
    MemberAuthor

    After some back and forth with an agent, we came up with this plan. This does not need to be the final plan, but it looks good to me as a starting point.

    Plan: Expose datafusion-spark Functions to Python

    Context

    The datafusion-spark Rust crate (in apache/datafusion at datafusion/spark/) provides ~74 scalar functions and 4 aggregate functions with Apache Spark-compatible semantics. We want to expose these in datafusion-python so users can opt into Spark-compatible behavior.

    The core challenge: ~14-15 spark functions share names with existing DataFusion built-ins but have different behavioral semantics. This affects both SQL (where functions are resolved by name from a registry) and the DataFrame API (where functions are bound at expression construction time).


    Name Collisions Inventory

    High-Impact Collisions (different results for same input)

    Function Spark Behavior DataFusion Behavior
    concat Returns NULL if any input is NULL Skips NULLs
    round HALF_UP rounding (2.5 → 3) Banker's rounding (2.5 → 2)
    date_part 1-indexed day-of-week (Sun=1) 0-indexed day-of-week
    date_trunc Converts to session TZ, always returns microseconds Preserves original precision, no TZ conversion
    substring/substr Negative start counts from end Different negative-index handling
    trunc Date truncation function Math truncation (toward zero) — completely different
    abs Returns min value on signed overflow Errors on overflow
    array_contains Three-valued NULL logic (NULL if element missing but array has NULLs) Returns false

    Moderate-Impact Collisions

    Function Difference
    avg (aggregate) Internal count uses i64 vs u64
    ascii Spark accepts numeric types via implicit cast
    factorial Spark restricts to Int32, returns NULL for out-of-range
    length/char_length Spark always returns Int32
    array_repeat Spark propagates NULL from count argument
    soundex Minor implementation differences
    array Different type coercion and null handling

    Non-Colliding Functions (~60, safe to add)

    add_months, base64, bin, bit_count, bit_get, bitmap_*, bitwise_not, char, collect_list, collect_set, crc32, csc, date_add, date_diff, date_sub, elt, expm1, format_string/printf, from_utc_timestamp, hex, hour, if, ilike, json_tuple, last_day, like, luhn_check, make_dt_interval, make_interval, map_from_entries, minute, mod, negative, next_day, parse_url, pmod, rint, sec, second, sha1/sha, sha2, shiftleft, shiftright, shiftrightunsigned, shuffle, size, slice, space, spark_cast, str_to_map, time_trunc, to_utc_timestamp, try_parse_url, try_sum, try_url_decode, unbase64, unhex, unix_date, unix_micros, unix_millis, unix_seconds, url_decode, url_encode, width_bucket


    Part 1: SQL Integration

    For SQL queries, function resolution happens through the SessionContext's function registry. The datafusion-spark crate provides register_all(&mut FunctionRegistry) which registers all spark functions, silently overwriting any existing function with the same name.

    Approach: ctx.enable_spark_functions() + selective ctx.register_spark_function(name)

    Add an explicit method on SessionContext, following the existing enable_url_table() pattern, plus a selective registration method for users who want fine-grained control:

    ctx = SessionContext()
    
    # Option 1: Register ALL spark functions (overwrites collisions)
    ctx.enable_spark_functions()
    
    # Option 2: Register a single spark function by name
    ctx.register_spark_function("round")
    
    # SQL now uses Spark semantics for registered colliding names
    result = ctx.sql("SELECT concat('a', NULL, 'b')").collect()
    # Returns NULL (Spark) instead of 'ab' (DataFusion)
    
    # Original DataFusion version still accessible via df_ prefix
    result = ctx.sql("SELECT df_concat('a', NULL, 'b')").collect()
    # Returns 'ab' (DataFusion)

    Implementation:

    • Rust (context.rs): Add enable_spark_functions(&self) that registers all spark functions, and register_spark_function(&self, name: &str) for selective registration. Both preserve originals under df_ prefix before overwriting.
    • Python (context.py): Thin wrappers calling the Rust methods.

    Preserving originals for SQL: Before overwriting a colliding function, re-register the original DataFusion function under a df_ prefix (e.g., df_concat, df_round). Users who need both semantics in one SQL query can use the prefixed name:

    SELECT concat('a', NULL, 'b'),    -- Spark: NULL
           df_concat('a', NULL, 'b')  -- DataFusion: 'ab'

    Selective registration is useful for users who only want specific spark behaviors (e.g., Spark's round semantics) without changing all colliding functions. The selective method also registers non-colliding spark-only functions (e.g., ctx.register_spark_function("crc32")).


    Part 2: DataFrame API Integration — Approaches

    For the DataFrame API, the situation is fundamentally different: Python functions like f.round(col("x")) bind directly to a function implementation at expression construction time (not via name lookup). So even if spark functions are registered on the context for SQL, datafusion.functions.round still uses the DataFusion implementation.

    Approach A: Separate datafusion.spark.functions Module (Recommended)

    All spark functions live in a dedicated namespace:

    from datafusion import functions as f
    from datafusion.spark import functions as sf
    
    # DataFusion semantics
    df.select(f.round(col("x"), lit(2)))       # banker's rounding
    
    # Spark semantics  
    df.select(sf.round(col("x"), lit(2)))      # HALF_UP rounding
    
    # Spark-only function (no collision)
    df.select(sf.crc32(col("data")))

    Pros:

    • Zero breaking changes — datafusion.functions is untouched
    • Explicit about which semantics you're using — no ambiguity
    • Clean IDE autocompletion: sf. shows all spark functions, f. shows all DF functions
    • Familiar to PySpark users (pyspark.sql.functions → datafusion.spark.functions)
    • Future-proof: same pattern works for datafusion.duckdb.functions etc.
    • All spark functions (colliding and non-colliding) live together consistently
    • Matches how the Rust crate is organized (datafusion-spark is its own crate)

    Cons:

    • Users who want spark-only functions (no collision) must import from the spark module
    • Puts "spark" in the module path — see naming discussion below
    • The ~60 non-colliding spark functions are less discoverable than if they were in datafusion.functions

    Approach B: Merge Everything into datafusion.functions

    Add all spark functions into the existing module. For collisions, use prefixed names (spark_round, spark_concat). Non-colliding functions are added without prefix.

    from datafusion import functions as f
    
    f.round(col("x"), lit(2))          # DataFusion
    f.spark_round(col("x"), lit(2))    # Spark
    f.crc32(col("data"))               # Spark-only, no prefix

    Pros:

    • Single import — everything under f.
    • Non-colliding spark functions immediately discoverable
    • No new module to learn about

    Cons:

    • Inconsistent naming: spark-only functions have no prefix, colliding ones do. Is f.crc32 a "spark function"? Is f.round not?
    • Pollutes autocomplete with spark_* variants interleaved with regular functions
    • Already a 6231-line file — grows further
    • spark_ prefix is ugly and doesn't match how the functions are named in Spark itself
    • No clean way to "import all spark functions" — they're mixed in

    Approach C: Config-Based Switching

    A global/session config changes what f.round(...) resolves to:

    ctx = SessionContext()
    ctx.set_function_variant("spark")
    df.select(f.round(col("x"), lit(2)))  # now uses Spark semantics

    Pros:

    • Minimal API change for users migrating from PySpark wholesale
    • Same code works for both variants

    Cons:

    • Dangerous implicit behavior: same code produces different results depending on hidden state
    • Makes code reviews/debugging harder — f.round(...) has no visible indication of which implementation
    • Thread safety concerns with global state
    • Mixed usage (needing both semantics in one query) is awkward
    • Violates "explicit is better than implicit" (Python Zen)
    • IDE/type checkers can't distinguish variants

    Naming Concern: "spark" in the codebase

    The user raised whether putting spark in module paths sends the wrong message.

    Considerations:

    • The upstream Rust crate is officially named datafusion-spark in the Apache DataFusion project — this isn't our invention
    • Users looking for Spark compatibility will search for "spark" — discoverability matters
    • Alternative names (extended, compat, extras) are vague and don't convey what the functions actually do
    • The substrait module already establishes a precedent for naming submodules after the external standard they integrate with

    Recommendation: Use spark in the module name. It's accurate, discoverable, and matches the upstream crate name.


    Recommended Approach Summary

    SQL: ctx.enable_spark_functions() — all-or-nothing, explicit opt-in, preserves originals under df_ prefix.

    DataFrame API: Approach A — datafusion.spark.functions as a separate module. This is the cleanest separation and matches Python idioms.


    Implementation Plan

    Step 1: Add datafusion-spark as a default-on dependency

    Files:

    • /Cargo.toml (workspace): Add datafusion-spark = { version = "53" } to [workspace.dependencies]
    • /crates/core/Cargo.toml: Add datafusion-spark = { workspace = true, optional = true } and feature spark = ["dep:datafusion-spark"]. Add "spark" to the default feature list: default = ["mimalloc", "spark"]
    • /pyproject.toml: Add "spark" to features = ["substrait"] → features = ["substrait", "spark"]

    The spark feature is enabled by default, matching the substrait precedent. Users building custom wheels can opt out with --no-default-features if needed.

    Step 2: Rust — Add enable_spark_functions() and register_spark_function() to PySessionContext

    File: crates/core/src/context.rs

    Add methods (gated with #[cfg(feature = "spark")]):

    #[cfg(feature = "spark")]
    const SPARK_COLLISIONS: &[&str] = &[
        "abs", "ascii", "avg", "concat", "date_part", "date_trunc",
        "factorial", "length", "round", "soundex", "substring", "trunc",
        "array_contains", "array_repeat", "array",
    ];
    
    #[cfg(feature = "spark")]
    fn preserve_original_if_collision(&self, name: &str) -> PyDataFusionResult<()> {
        if SPARK_COLLISIONS.contains(&name) {
            if let Ok(original) = self.ctx.udf(name) {
                let renamed = original.as_ref().clone().with_name(format!("df_{name}"));
                self.ctx.register_udf(renamed);
            }
        }
        Ok(())
    }
    
    #[cfg(feature = "spark")]
    pub fn enable_spark_functions(&self) -> PyDataFusionResult<()> {
        // Save originals under df_ prefix for all colliding functions
        for name in SPARK_COLLISIONS {
            self.preserve_original_if_collision(name)?;
        }
        // Register all spark functions
        for udf in datafusion_spark::all_default_scalar_functions() {
            self.ctx.register_udf((*udf).clone());
        }
        for udaf in datafusion_spark::all_default_aggregate_functions() {
            self.ctx.register_udaf((*udaf).clone());
        }
        Ok(())
    }
    
    #[cfg(feature = "spark")]
    pub fn register_spark_function(&self, name: &str) -> PyDataFusionResult<()> {
        // Look up the spark function by name across all function types
        let all_scalars = datafusion_spark::all_default_scalar_functions();
        if let Some(udf) = all_scalars.iter().find(|f| f.name() == name) {
            self.preserve_original_if_collision(name)?;
            self.ctx.register_udf((**udf).clone());
            return Ok(());
        }
        let all_aggs = datafusion_spark::all_default_aggregate_functions();
        if let Some(udaf) = all_aggs.iter().find(|f| f.name() == name) {
            self.preserve_original_if_collision(name)?;
            self.ctx.register_udaf((**udaf).clone());
            return Ok(());
        }
        Err(PyValueError::new_err(
            format!("No spark function named '{name}' found")
        ).into())
    }

    File: python/datafusion/context.py — Add Python wrappers:

    def enable_spark_functions(self) -> None:
        """Register all Spark-compatible functions on this context.
    
        Colliding functions are overwritten with Spark semantics.
        Original DataFusion versions remain accessible with a ``df_`` prefix
        (e.g., ``df_round``, ``df_concat``).
        """
        self.ctx.enable_spark_functions()
    
    def register_spark_function(self, name: str) -> None:
        """Register a single Spark-compatible function by name.
    
        If the function collides with an existing DataFusion built-in,
        the original is preserved with a ``df_`` prefix.
    
        Args:
            name: The name of the spark function to register (e.g., "round", "crc32").
    
        Raises:
            ValueError: If no spark function with the given name exists.
        """
        self.ctx.register_spark_function(name)

    Step 3: Rust — Create spark_functions.rs module

    File: crates/core/src/spark_functions.rs (new)

    Define #[pyfunction] wrappers for each spark function. For each of the ~74 scalar functions:

    #[pyfunction]
    fn round(value: PyExpr, decimal_places: Option<PyExpr>) -> PyExpr {
        // Use the spark UDF directly to create the expression
        let spark_round_udf = datafusion_spark::spark_round();  // or equivalent accessor
        Expr::ScalarFunction(ScalarFunction::new_udf(spark_round_udf, args)).into()
    }

    For the 4 aggregate functions, follow the add_builder_fns_to_aggregate pattern from functions.rs.

    Define pub(crate) fn init_module(m: &Bound<'_, PyModule>) -> PyResult<()> that registers all functions.

    Note: The exact API for constructing expressions from datafusion-spark UDFs needs verification against the crate's public API. It may provide expr_fn::round(...) convenience functions or require ScalarFunction::new_udf().

    Step 4: Rust — Register spark_functions submodule

    File: crates/core/src/lib.rs

    #[cfg(feature = "spark")]
    pub mod spark_functions;
    
    // In init_module:
    #[cfg(feature = "spark")]
    {
        let spark_funcs = PyModule::new(py, "spark_functions")?;
        spark_functions::init_module(&spark_funcs)?;
        m.add_submodule(&spark_funcs)?;
    }

    Step 5: Python — Create datafusion.spark package

    Files:

    • python/datafusion/spark/__init__.py — Minimal, exports functions
    • python/datafusion/spark/functions.py — Python wrappers for all spark functions

    Pattern for each function:

    from datafusion._internal import spark_functions as sf
    
    def round(value: Expr, decimal_places: Expr | None = None) -> Expr:
        """Round using Spark HALF_UP semantics.
    
        Unlike :py:func:`datafusion.functions.round` which uses banker's rounding,
        this rounds half values away from zero (2.5 → 3).
    
        Examples:
            >>> ...
        """
        return Expr(sf.round(value.expr, decimal_places.expr if decimal_places else None))

    Each colliding function's docstring should note the behavioral difference from the DataFusion built-in.

    File: python/datafusion/__init__.py — Add spark to imports and __all__.

    Step 6: Handle feature flag gracefully in Python

    In python/datafusion/spark/__init__.py:

    try:
        from datafusion._internal import spark_functions as _sf
        SPARK_AVAILABLE = True
    except ImportError:
        SPARK_AVAILABLE = False

    Functions in spark/functions.py raise a clear error if the feature isn't compiled.

    Step 7: Testing

    File: python/tests/test_spark_functions.py (new)

    • Test all spark functions for correctness
    • For colliding functions, add comparison tests demonstrating the behavioral difference:
      def test_round_spark_vs_default():
          # DataFusion uses banker's rounding
          assert f.round(lit(2.5)).collect() == 2  # round to even
          # Spark uses HALF_UP
          assert sf.round(lit(2.5)).collect() == 3  # round away from zero
    • Test enable_spark_functions() works for SQL queries
    • Test register_spark_function("round") registers only that one function
    • Test that register_spark_function("nonexistent") raises ValueError
    • Test that df_ prefixed originals are accessible after enabling spark functions

    Step 8: Documentation

    • Add docs page at docs/source/user-guide/spark-functions.rst explaining the module
    • Document all collisions with behavioral differences
    • Add migration guide for PySpark users

    Verification Plan

    1. Build: maturin develop compiles successfully (spark enabled by default)
    2. Import test: from datafusion.spark import functions as sf works
    3. SQL test (enable all): After ctx.enable_spark_functions(), SELECT round(2.5) returns 3 (Spark HALF_UP)
    4. SQL test (selective): After ctx.register_spark_function("round"), SELECT round(2.5) returns 3 but SELECT concat('a', NULL, 'b') still returns 'ab' (DataFusion)
    5. SQL test (selective error): ctx.register_spark_function("nonexistent") raises ValueError
    6. DataFrame test: sf.round(lit(2.5)) returns 3, f.round(lit(2.5)) returns 2
    7. Originals preserved: After enabling, SELECT df_round(2.5) returns 2
    8. Non-colliding functions: sf.crc32(lit(b"hello")) works
    9. Feature flag off: Without spark feature, from datafusion.spark import functions raises clear ImportError
    10. Existing tests pass: No regressions in python/tests/

    Key Files to Modify/Create

    File Action
    /Cargo.toml Add datafusion-spark to workspace deps
    /crates/core/Cargo.toml Add optional dep + spark feature
    /pyproject.toml Add "spark" to maturin features
    /crates/core/src/context.rs Add enable_spark_functions()
    /crates/core/src/spark_functions.rs New — Rust PyO3 wrappers for all spark functions
    /crates/core/src/lib.rs Register spark_functions submodule
    /python/datafusion/spark/__init__.py New — Package init
    /python/datafusion/spark/functions.py New — Python wrappers with docstrings
    /python/datafusion/__init__.py Add spark to exports
    /python/datafusion/context.py Add enable_spark_functions() wrapper
    /python/tests/test_spark_functions.py New — Tests
  3. djouallah commented on Apr 11, 2026

    @djouallah
    Contributor

    How about standardizing on spark API as the default, there is really no point in the long term to have two API ?

  4. timsaucer commented on Apr 11, 2026

    @timsaucer
    MemberAuthor

    How about standardizing on spark API as the default, there is really no point in the long term to have two API ?

    I don't think we'd want to standardize on spark as default, especially since some functions return differently. Then anyone who is expected PostgreSQL-like queries to work would fail / return unexpected results. Do you disagree?

    It feels to me like enabling spark functions in the SQL syntax needs to be an opt-in feature.

  5. djouallah commented on Apr 12, 2026

    @djouallah
    Contributor

    I was referring more to the syntax, since it makes it easier for both humans and agents to write familiar code. To be clear, I don’t think matching Spark’s output is a good idea, especially since even Spark is moving toward ANSI compliance by default.

  6. timsaucer commented on Apr 12, 2026

    @timsaucer
    MemberAuthor

    I was referring more to the syntax, since it makes it easier for both humans and agents to write familiar code. To be clear, I don’t think matching Spark’s output is a good idea, especially since even Spark is moving toward ANSI compliance by default.

    I'm sorry, but I don't understand what you're specifically recommending.

  7. djouallah commented on Apr 12, 2026

    @djouallah
    Contributor

    it will be nice maybe to use the same spark api syntax, even if the output does not 100 % match the expected behavior of Spark

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions