Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide - #578
Open
rjurney wants to merge 107 commits into
Open
Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide#578rjurney wants to merge 107 commits into
rjurney wants to merge 107 commits into
Conversation
…s.txt and split out requirements-dev.txt. Version bumps.
…ney/build-upgrades
…ney/build-upgrades
|
|
||
| Before starting this tutorial, ensure you have: | ||
|
|
||
| - **GraphFrames installed**: `pip install graphframes-py` |
Collaborator
There was a problem hiding this comment.
+ graphframes core
| Before starting this tutorial, ensure you have: | ||
|
|
||
| - **GraphFrames installed**: `pip install graphframes-py` | ||
| - **Apache Spark 3.x**: Compatible with your Python version |
Collaborator
There was a problem hiding this comment.
+ spark should be compatible with GF version
| </figure> | ||
| </center> | ||
|
|
||
| ## Prerequisites |
Collaborator
There was a problem hiding this comment.
I would point people to installation section of docs
| # Create a GraphFrame to get access to AggregateMessages API | ||
| g: GraphFrame = GraphFrame(nodes_df, edges_df) | ||
|
|
||
| msgToDst = AM.src["start_degree"] |
Collaborator
There was a problem hiding this comment.
Why not just lit(1)?
| g_simple = GraphFrame(vertices_simple, edges_simple) | ||
|
|
||
| # Calculate in-degree using Pregel API | ||
| pregel_result = g_simple.pregel \ |
Collaborator
There was a problem hiding this comment.
It is better to wrap chains to () instead of using line breaks imo. And in previous snippets you were using ()
| # +---+-------+---------+ | ||
| ``` | ||
|
|
||
| ### Understanding the Pregel API |
Collaborator
There was a problem hiding this comment.
I was expecting you will highlight the whole API... Early stopping conditions especially. For example, PageRank may be run for a fixed num of iterations, but also until convergence based on tolerance factor. I was thinking you put it to the end, but you didn't. Can you add a couple of words about different early stopping strategies and maybe a simple example?
Collaborator
There was a problem hiding this comment.
That is the docs section, jfyi: https://graphframes.io/04-user-guide/10-pregel.html#termination-conditions
| F.coalesce(Pregel.msg(), Pregel.src("label"))) \ | ||
| .sendMsgToDst(Pregel.src("label")) \ | ||
| .sendMsgToSrc(Pregel.dst("label")) \ | ||
| .aggMsgs(F.expr("mode(collect_list(msg))")) \ |
Collaborator
There was a problem hiding this comment.
And there is a small tricky story :)
Deterministic versus not. To pass LDBC tests you need to run mode with deterministic=True (was added in spark 4.x)
Collaborator
There was a problem hiding this comment.
Actually that is the reason why GF implementation does not use mode (tldr we will switch to mode after spark 3.5.x EOL)
…y/pregel-tutorial
…mes into rjurney/pregel-tutorial
rjurney
changed the title
…ake sure it all works. Update codespell for Sergey Brin.
… problem with pandas/numpy version 2.5.1 being required by Python >= 3.10
…ontext lessons Present the original broken LPA code as a teaching device: explain why Pregel.src() fails outside message expressions, why aggMsgs forbids nested aggregates (mode(collect_list()) -> F.mode(Pregel.msg())), and why withVertexColumn must own the label column. Add verified corrected implementation with real Stack Exchange output and a log2-bucketed text histogram of community sizes with log10-scaled bars.
…-weighted PageRank
Replace simulated Id % 3 node types with the graph's real Type column and
fix the type weight to use Pregel.src("Type") in the message expression
(triplet context). Add human-readable names via coalesce over DisplayName/
Title/TagName/Body, verified output tables, and a new subsection explaining
the User teleport-floor result: edges point from users, so rank never flows
to them - edge direction determines which node types accumulate importance.
Add "suppor" to codespell ignore list: it appears in verbatim truncated
Spark output quoted in the tutorial.
…y/pyproject-numpy # Conflicts: # python/poetry.lock # python/pyproject.toml
…mes into rjurney/pregel-tutorial
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…mes into rjurney/pregel-tutorial
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds end-to-end tutorial material for GraphFrames’ Pregel API and a Neo4j ↔ GraphFrames integration workflow, including new Python tutorial scripts/CLI commands and new documentation pages to guide users through data preparation, loading into Neo4j, running GraphFrames algorithms, and writing results back.
Changes:
- Add a new Neo4j integration tutorial (docs + Python tutorial scripts +
graphframes neo4j ...CLI). - Add a new Pregel tutorial to the docs and update spellcheck configuration accordingly.
- Update Python packaging (Poetry) to include runtime/tutorial dependencies needed by the new tutorial code.
Reviewed changes
Copilot reviewed 13 out of 18 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| python/tests/test_neo4j_tutorial.py | Adds a Neo4j tutorial “validation” script (currently placed under tests). |
| python/pyproject.toml | Adds/updates Python dependencies for runtime and tutorial/dev groups. |
| python/poetry.lock | Updates locked Python dependency set to match pyproject changes. |
| python/graphframes/tutorials/stackexchange.py | Adjusts data path resolution and writes additional Parquet outputs for Neo4j loading. |
| python/graphframes/tutorials/neo4j/loaders.py | Adds APOC Parquet loading helpers and loader registry for node/edge types. |
| python/graphframes/tutorials/neo4j/load.py | Adds a Spark-based Neo4j loading script (connector-based alternative). |
| python/graphframes/tutorials/neo4j/docker.py | Adds helpers to start/stop/manage a Neo4j Docker container configured for APOC. |
| python/graphframes/tutorials/neo4j/connected_components.py | Adds a script to read from Neo4j, run Connected Components, and write results back. |
| python/graphframes/tutorials/neo4j/cli.py | Adds graphframes neo4j CLI group for container management, loading, and status. |
| python/graphframes/tutorials/neo4j/init.py | Exposes the Neo4j CLI entry point from the tutorials package. |
| python/graphframes/console.py | Registers the new Neo4j CLI command in the main graphframes CLI. |
| docs/src/03-tutorials/04-neo4j-integration.md | Adds detailed Neo4j integration tutorial documentation. |
| docs/src/03-tutorials/03-pregel-tutorial.md | Adds a comprehensive Pregel tutorial page. |
| .codespellrc | Updates codespell ignore list to accommodate tutorial text. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
1
to
15
| @@ -10,6 +11,7 @@ def cli(): | |||
|
|
|||
|
|
|||
| cli.add_command(download.stackexchange) | |||
| cli.add_command(neo4j) | |||
|
|
|||
Comment on lines
+18
to
+37
| # Compute the default data directory relative to this package | ||
| _PACKAGE_DIR = Path(__file__).parent.parent # tutorials/ | ||
| DEFAULT_DATA_DIR = str(_PACKAGE_DIR / "data") | ||
|
|
||
| from graphframes.tutorials.neo4j.docker import ( | ||
| CONTAINER_NAME, | ||
| DEFAULT_PASSWORD, | ||
| check_docker, | ||
| container_exists, | ||
| remove_container, | ||
| start_container, | ||
| wait_for_ready, | ||
| ) | ||
| from graphframes.tutorials.neo4j.loaders import ( | ||
| EDGE_LOADERS, | ||
| NODE_LOADERS, | ||
| clear_database, | ||
| create_indexes, | ||
| get_parquet_file_uris, | ||
| ) |
Comment on lines
+1
to
+12
| """ | ||
| Test script for Neo4j integration tutorial. | ||
| Tests the data processing logic without requiring Neo4j. | ||
| """ | ||
| import pyspark.sql.functions as F | ||
| from graphframes import GraphFrame | ||
| from pyspark.sql import SparkSession, DataFrame | ||
|
|
||
| print("=" * 80) | ||
| print("NEO4J TUTORIAL - CODE VALIDATION") | ||
| print("=" * 80) | ||
|
|
Comment on lines
+6
to
+10
| spark-submit \\ | ||
| --packages org.neo4j:neo4j-connector-apache-spark_2.12:5.3.1_for_spark_3 \\ | ||
| --driver-memory 4g \\ | ||
| --executor-memory 4g \\ | ||
| python/graphframes/tutorials/neo4j_load.py |
Comment on lines
+5
to
+9
| spark-submit \\ | ||
| --packages org.neo4j:neo4j-connector-apache-spark_2.12:5.3.1_for_spark_3 \\ | ||
| --driver-memory 4g \\ | ||
| --executor-memory 4g \\ | ||
| python/graphframes/tutorials/neo4j_connected_components.py |
| | name| pagerank| | ||
| +--------------------------------------------------+--------------------+ | ||
| | TeX processing for Stats| 0.1450891970426848| | ||
| |What typographic support is available to suppor...| 0.14112968883288293| |
Comment on lines
+12
to
+14
| # brin - Sergey Brin, co-founder of Google (PageRank co-inventor) | ||
| # suppor - truncated word in verbatim Spark output in the Pregel tutorial | ||
| ignore-words-list = mis,te,ser,fof,mye,protocall,alledges,afterall,jurney,brin,suppor |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?