Issue
Using a Microsoft Fabric notebook, running the Breadth-First Search function of a graphframe returns a Py4JError during execution when using the Pyspark version of the library.

To Reproduce

Steps to reproduce the behavior:

1. Open Microsoft Fabric Notebook with an environment attached to it
2. Install the .whl package via magic command (using the file in the environment or the file from pypi)
3. Install the Jar file in the spark.jars variable of the config (this can be done inside the environment itself)
4. Execute the following example code from the Documentation

    from graphframes.examples import Graphs
    
    g = Graphs(spark).friends()  # Get example graph

    # Search from "Esther" for users of age < 32

    paths = g.bfs("name = 'Esther'", "age < 32")
    paths.show()

    # Specify edge filters or max path lengths

    g.bfs("name = 'Esther'", "age < 32",
        edgeFilter="relationship != 'friend'", maxPathLength=3)

Expected behavior

Code Should return a Dataframe with the results

System [please complete the following information]:

• PySpark version: PySpark 3.5
• Scala version: Scala 2.12.17
• Python version: 3.11
• GraphFrames version: graphframes-0.10.0
• GraphFrames JAR version: graphframes-spark3_2.12-0.10.0.jar

Component

• Scala Core Internal
• Scala API
• Spark Connect Plugin
• PySpark Classic
• PySpark Connect

Additional context

Running the Scala Example for the same function returns a proper result.

    %%spark
    import org.graphframes.{examples, GraphFrame}

    val g: GraphFrame = examples.Graphs.friends // get example graph

    // Search from "Esther" for users of age < 32.
    val paths = g.bfs.fromExpr("name = 'Esther'").toExpr("age < 32").run()
    paths.show()

    // Specify edge filters or max path lengths.
    val paths = {
    g.bfs.fromExpr("name = 'Esther'").toExpr("age < 32")
        .edgeFilter("relationship != 'friend'")
        .maxPathLength(3).run()
    }
    paths.show()

Error_trace.txt

Graphframes_Error_Example.ipynb