doc: Pregel Tutorial by rjurney · Pull Request #809 · graphframes/graphframes

rjurney · 2026-03-13T06:47:19Z

New, big, fancy, super, duper Pregel tutorial
Moved Stack Exchange data content from Network Motif Finding Tutorial into Data Setup tutorial. Refer to from both motif and Pregel tutorials.
Point at new tutorial(s) from list of tutorials.
New network motif and Pregel tutorial Jupyter notebooks
Some other minor changes...

…s.txt and split out requirements-dev.txt. Version bumps.

…ney/build-upgrades

…llback key names

Copilot

Pull request overview

Copilot reviewed 32 out of 56 changed files in this pull request and generated 1 comment.

Suppressed comments (7)

Previously missed (4) — in code that hasn't changed since the last review.

python/pyproject.toml:97

neo4j is added to the tutorials dependency group, but there are no imports/uses of the Neo4j Python driver in the tutorials (they use the Neo4j Spark connector via JVM packages and Docker). Keeping an unused dependency increases install time and supply-chain surface area; remove it until it’s actually needed.
docs/src/03-tutorials/05-neo4j-integration.md:138
Duplicate word: “creates the the …”.

The `graphframes neo4j load` command creates the the nodes, edges and indices — and prints their verification counts at the end. Point it elsewhere with `--data-dir`, `--site` and the `--neo4j-*` connection options.

docs/src/03-tutorials/05-neo4j-integration.md:231

Typo in section heading: “Read the gG”.

### Read the gG

docs/src/03-tutorials/03-data-setup.md:129

This code comment implies the Parquet contains both Id and id, but python/graphframes/tutorials/stackexchange.py renames Id to StackId and drops Id before writing Nodes.parquet. Consider rewording to a more general explanation of why case sensitivity may be needed (or reference StackId instead of Id).

# Lets the Id:(Stack Overflow int) and id:(GraphFrames ULID) coexist
spark.conf.set("spark.sql.caseSensitive", True)

python/graphframes/tutorials/neo4j.py:31

The script hard-codes Neo4j credentials (including a password) in source. This is risky to ship as-is and also makes the tutorial brittle if the user ran graphframes neo4j setup --password ... with a different password (the subsequent spark-submit will fail to connect). Prefer reading connection settings from environment variables (or CLI args) with safe defaults for the tutorial.
.pre-commit-config.yaml:40
The codespell hook exclude regex targets python/tutorials/, but this repo’s tutorial code lives under python/graphframes/tutorials/ (and python/tutorials/ doesn’t exist). As written, the exclude won’t match and codespell will still scan the tutorial tree.

    hooks:
      - id: codespell
        exclude: ^(graphx|python/tutorials/)

docs/src/helium/custom.css:16

figcaption text is forced to white (#fff), which can become unreadable on Helium’s default light backgrounds and reduces contrast for accessibility. Let the caption inherit the theme text color (or choose a dark color) instead of hard-coding white.

figcaption {
    display: block;
    text-align: center;
    font-size: 0.875em;
    color: #fff;
    font-style: italic;

rjurney · 2026-08-29T08:37:51Z

 import logging

+from graphframes.tutorials import download
+


cc @SemyonSinchenko plz to review

james-willis · 2026-08-31T16:48:19Z

-This tutorial covers GraphFrames' motif finding feature. We perform pattern matching on a property graph representing a Stack Exchange site using Apache Spark and [GraphFrames' motif finding](/04-user-guide/04-motif-finding.md) feature. We will download the `stats.meta` archive from the [Stack Exchange Data Dump at the Internet Archive](https://archive.org/details/stackexchange), use PySpark to build a property graph and then mine it for property graph network motifs by combining both graph and relational queries.
+This tutorial covers GraphFrames' motif finding feature using **Apache Spark 4.x** and [GraphFrames' motif finding](/04-user-guide/04-motif-finding.md). We perform pattern matching on a property graph representing a Stack Exchange site, using PySpark to build a property graph and then mine it for property graph network motifs by combining both graph and relational queries.
+
+A Jupyter Notebook version of this tutorial is available on GitHub: [Network Motif Finding Notebook](https://github.com/graphframes/graphframes/blob/master/python/graphframes/tutorials/notebooks/Network_Motif_Finding.ipynb).


I think you deleted these notebooks

Thanks, will fix.

james-willis · 2026-08-31T16:48:41Z

        class RWEmbeddings(LogicalPlan):
            def __init__(
-                self, v: DataFrame, e: DataFrame, params: _RandomWalksEmbeddingsParameters
+                self,


formattter churn is frustrating in a 5k+ line PR

Yeah, and yet most of the code is unformatted a lot of the time. How does this shit get through?

Obviously its because it is not enforced in the CI.

I'll raise a PR to make python linting enforced in the CI but in the future you can do this yourself if it bothers you so much.

You seem to be running black with a different config than the repo uses.

We are using line length 100 but maybe you are using 88?

you are linting test dirs, doc dirs, and generated files. These I fix in ci: run python lint on tests/ and docs/, as its own job #896

Line length is configured to 100 in the repo, is it not?

[tool.black] line-length = 100 target-version = ["py39"] include = ["graphframes"]

Oh, I'm using 88? I'll check...

python/graphframes/connect/proto/ is generated by `buf generate` (see buf.gen.yaml, which runs the protocolbuffers/pyi and python plugins into that directory). Formatting generated output fights the generator: black reflows it, and the next regen emits the original again, so the same churn reappears on every regeneration. This already bit graphframes#809, where graphframes_pb2.pyi picked up a reformatted __slots__ tuple that has nothing to do with that PR. black needs force-exclude rather than exclude, because exclude is not applied to paths passed explicitly on the command line, and both CI and the pre-commit hook pass paths explicitly. flake8 is left alone: tox.ini already carves out E501/F401 for this directory, and its remaining checks do not rewrite the file.

* ci: run python lint on tests/ and docs/, as its own job The Python style check already existed, but only covered `graphframes`, `python/dev` and `dev`. `python/tests` and `python/docs` were never format-checked, so formatting churn in those directories accumulated on main and surfaced later inside unrelated feature PRs. `python/tests` is touched by most feature PRs, which is where this is felt most. Changes: - Add `tests` and `docs` to the checked paths, in both the CI workflow and .pre-commit-config.yaml, and note that the two lists must agree. - Move the style check out of the Spark test matrix into a standalone `lint` job. It ran three times (once per matrix entry) behind a full JDK/Spark/jar setup; now it runs once and reports independently. Each tool is its own step so a failure names the tool. - Split black/isort/flake8 into separate steps for clearer failures. - Fix `[tool.black] include`, which was a list where black expects a regex string. A list silently disables black's default `\.pyi?$` filter, so black tried to parse Makefile/.rst/.css when handed a directory. This blocked linting `docs/` at all. - Apply black + isort to the newly covered files, and fix the six remaining flake8 findings (3x E265, 2x E501, 1x W391). Everything outside the two newly-covered directories is untouched; `graphframes`, `dev` and `../dev` were already clean and stay byte identical. * ci: exclude generated protobuf stubs from black and isort python/graphframes/connect/proto/ is generated by `buf generate` (see buf.gen.yaml, which runs the protocolbuffers/pyi and python plugins into that directory). Formatting generated output fights the generator: black reflows it, and the next regen emits the original again, so the same churn reappears on every regeneration. This already bit #809, where graphframes_pb2.pyi picked up a reformatted __slots__ tuple that has nothing to do with that PR. black needs force-exclude rather than exclude, because exclude is not applied to paths passed explicitly on the command line, and both CI and the pre-commit hook pass paths explicitly. flake8 is left alone: tox.ini already carves out E501/F401 for this directory, and its remaining checks do not rewrite the file.

SemyonSinchenko · 2026-09-01T10:36:11Z

Note: we have validate method for this.

SemyonSinchenko · 2026-09-01T10:41:57Z

Are we 100% sure we can use Neo4j logo here?

SemyonSinchenko · 2026-09-01T10:43:51Z

Note: tbh I cannot get why for mapping to neo4j you used GraphFrame instead of PropertyGraphFrame. The second suits better imo because it has explicit schema.

SemyonSinchenko · 2026-09-01T10:44:22Z

Note: from what I see around the world, algorithms are running on projections, not whole LPG.

SemyonSinchenko

~LGTM overall. I'm ready to approve after merge-conflicts are addressed.

rjurney · 2026-09-01T20:28:24Z

Note: tbh I cannot get why for mapping to neo4j you used GraphFrame instead of PropertyGraphFrame. The second suits better imo because it has explicit schema.

This is great feedback, let me fix it. I've just never used PropertyGraphFrame.

rjurney · 2026-09-01T20:28:58Z

Are we 100% sure we can use Neo4j logo here?

I assumed they wouldn't mind but I don't know. Let me ask permission.

rjurney · 2026-09-01T20:29:16Z

Note: we have validate method for this.

Thanks, will integrate.

@SemyonSinchenko

Address review feedback from @SemyonSinchenko on PR graphframes#809: - Read the Neo4j export into an explicit PropertyGraphFrame (one VertexPropertyGroup for nodes, one EdgePropertyGroup per relationship type) instead of building a flat GraphFrame directly from two untyped Cypher reads. - Project that property graph to a GraphFrame via to_graphframe(), and explain the projection concept: this call always takes an explicit list of vertex/edge groups, so it doubles as documentation of which parts of the graph feed the algorithm that follows. - Call graph.validate() after the projection to catch duplicate vertex ids or dangling edges up front, per @SemyonSinchenko's note that GraphFrame already has a validate() method for this. - Bring Type back onto the Connected Components output with an explicit join on id, since to_graphframe() only carries id and property_group by design. The resulting graph, edges and Connected Components grouping are identical to the previous flat-GraphFrame version (verified locally against synthetic multi-type data), so the rest of the tutorial - the write-back, the demo query, and all of the specific counts - is unchanged. Co-authored-by: Russell Jurney <[email protected]>

@james-willis

…'s 100 @james-willis and @rjurney found that graphframes_client.py had been reformatted with black's default 88-column width instead of the repo's configured line-length = 100 (pyproject.toml), producing avoidable diff churn on lines that fit on one line at 100 columns. Audited every file this PR touches for the same mistake (a line 90-100 characters wide, force-split onto multiple lines with a trailing comma that then "sticks" under black's magic-trailing-comma rule) and collapsed each one back to what black --config pyproject.toml (line-length 100, version 23.12.1, the pinned required-version) actually produces: - graphframes/connect/graphframes_client.py: RWEmbeddings.__init__ - graphframes/graphframe.py: triangleCount() signature and call - graphframes/pg/property_graphframe.py: the e1.join(e2, ...) call - graphframes/tutorials/{pregel,neo4j_cli,generate_diagrams,motif}.py python/docs/{conf,epytext,underscores}.py, python/tests/conftest.py, python/tests/test_graphframes.py and python/tests/pg/test_property_graphframe.py were independently reformatted by both this branch and the already-merged graphframes#896 ("run python lint on tests/ and docs/"); realigned them to be byte-identical to main's post-graphframes#896 versions (confirmed via diff) to remove that merge-conflict surface entirely. graphframes/connect/proto/graphframes_pb2.pyi is generated by the buf tool; reverted its accidental reformatting back to the raw generator output, matching main, since graphframes#896 also force-excludes that directory from black/isort for the same reason. Every change here is formatting-only: verified with an AST-equality check (old vs. new) for each modified file, plus a full local test run (python/tests, 204 tests, all passing) and a clean black/isort/flake8 pass across graphframes/, tests/, docs/, dev/ and ../dev/. Co-authored-by: Russell Jurney <[email protected]>

Follow-up to the PropertyGraphFrame refactor: model each of the seven real node types (User, Badge, Vote, Question, Answer, PostLinks, Tag) as its own VertexPropertyGroup instead of lumping them into a single "nodes" group, matching the "Shape" column of the tutorial's own relationship table one EdgePropertyGroup at a time. CastFor, Tags, Links and Duplicates all connect to a Post - Stack Exchange's own term for "a Question or an Answer" - and EdgePropertyGroup needs one fixed vertex group per side, so those four relationship types reference a small "post" VertexPropertyGroup (the union of Question and Answer) purely for that routing. Since ids are unmasked, "post" shares the exact same id space as "questions"/ "answers" and is never itself included in a vertex projection. Naming every real-type group after its Type value means to_graphframe()'s "property_group" column already *is* Type once projected, so recovering it after Connected Components is now a plain column rename instead of a join back onto the original `vertices` DataFrame - simpler than the previous version. Verified with synthetic data covering every node type and every relationship shape, including the two-ends-ambiguous Links/Duplicates case: vertices, edges and the Connected Components grouping are identical to both the old single-group PropertyGraphFrame version and the original flat GraphFrame(vertices, edges) approach. Also ran the exact markdown code blocks end-to-end the same way. black/isort/flake8 pass, and the full local test suite (109 tests) still passes. Co-authored-by: Russell Jurney <[email protected]>

…lack-fmt-8c8c docs(neo4j): use PropertyGraphFrame + validate(), and fix black line-length-88 formatting churn

…oordinates (#10) * fix(python): repair corrupted pyproject.toml on rjurney/pypi-tutorials `poetry install`/`poetry check` currently fail outright on this branch: Invalid TOML file /workspace/python/pyproject.toml: Unexpected end of file at line 137 col 0 The [tool.black] section has two leftover merge-conflict remnants - a stray `'''` and `=======` - left behind by "Merge branch 'main' into rjurney/pypi-tutorials" (37df7b9), making the file invalid TOML. Removed both stray lines. That still left `force-exclude` combining main's proto exclusion with this branch's tutorials-data exclusion as a Python list - `["/graphframes/connect/proto/", "/graphframes/tutorials/data/"]` - which is the same mistake `include` had before graphframes#896 fixed it: black's own `--verbose` output confirms a list there matches *every* file ("ignored: matches the --force-exclude regular expression" on files that match neither string), not just the two intended paths. `force-exclude` needs a single regex string. Combined both exclusions into one string with alternation instead. Verified on a clean checkout of origin/rjurney/pypi-tutorials: - `poetry check` and `poetry install --with=dev` both now succeed (they currently fail with exit code 1 on this branch as-is). - `python -c "import tomllib; tomllib.load(...)"` parses the file. - `black --check` now finds and correctly formats all 40 real Python files under graphframes/tests/docs/dev, while still excluding the generated graphframes_pb2.pyi and everything under graphframes/tutorials/data/. - `isort --check` and `flake8` both clean. - `poetry check --lock` clean (no lock-file drift from this fix). - Full local test suite: 109/109 passing. Co-authored-by: Russell Jurney <[email protected]> * fix(neo4j): use the Neo4j connector's actual current Maven coordinates Ran the full tutorial for real to check it: downloaded the real stats.meta.stackexchange.com dump, built Nodes/Edges.parquet, stood up a real Neo4j server (no Docker available, so the plain tarball distribution instead - same Bolt/HTTP endpoints `graphframes neo4j setup` would give you), ran `graphframes neo4j load` against it, then spark-submit'd neo4j.py for real. That surfaced two real bugs in the documented Maven coordinates, both invisible to a plain dependency resolution check because Ivy silently follows Maven relocation POMs: 1. `org.neo4j:neo4j-connector-apache-spark_2.13:6.0.0_for_spark_4` is a relocation stub - Neo4j moved the connector to `org.neo4j.connectors:spark` at 6.0.0. It still resolves and runs, but points at a coordinate Neo4j's own docs no longer document, so switched every reference to the live one: `org.neo4j.connectors:spark:6.0.0-s_2.13`. 2. `6.0.0_for_spark_3` relocates to the exact same Spark-4-only jar as `6.0.0_for_spark_4` - the message on the relocation POM says so outright ("requires Spark 4 with Scala 2.13") - so the tutorial's Spark 3.5 guidance was actually pointing Spark 3.5 users at a jar that cannot load under Spark 3.5. Confirmed the failure directly (`NoClassDefFoundError: scala/$less$colon$less`, a Scala 2.13-vs-2.12 class loaded under a Scala 2.12 Spark 3.5 runtime) and confirmed the fix: `graphframes-spark3_2.12:0.12.1` (PySpark 3.5 ships Scala 2.12, not the 2.13 the tutorial suggested) with `neo4j-connector-apache-spark_2.12:5.4.3_for_spark_3`, the last connector line that actually targets Spark 3.5. Also aligned `neo4j_cli.py`'s NEO4J_PACKAGE constant (used for `load`'s own SparkSession, built outside spark-submit) to the same live coordinate, and noted there that `load` specifically needs a Spark 4.x PySpark install regardless of which Spark version `neo4j.py` targets. Verified end to end, twice, against the real Neo4j instance with the real 129,751-node/97,104-edge graph, using the exact corrected commands: - Spark 4.1.3 (poetry's pinned dev dependency), packages `graphframes-spark4_2.13:0.12.1,org.neo4j.connectors:spark:6.0.0-s_2.13`: `graphframes neo4j load` then `spark-submit .../neo4j.py` both ran clean, producing the exact counts the tutorial documents (40,115 components, the 56,442-node/6-type giant-component breakdown, the same top-10 decorated users by badge count), and the write-back actually landed - every one of the 129,751 nodes came back out of Neo4j with a `component` property afterward. - Spark 3.5.8 in an isolated venv (PySpark's actual bundled Scala, 2.12), packages `graphframes-spark3_2.12:0.12.1,org.neo4j:neo4j-connector-apache-spark_2.12:5.4.3_for_spark_3`: same script, same real Neo4j data, identical results. Confirmed the currently-documented (pre-fix) coordinates actually fail under Spark 3.5 by running them as-is first and reproducing the NoClassDefFoundError, then confirmed the fix resolves it - not just that the new coordinate exists on Maven Central. black/isort/flake8 clean; full local suite (109 tests) still passes. Co-authored-by: Russell Jurney <[email protected]> --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Russell Jurney <[email protected]>

SemyonSinchenko

~LGTM

rjurney added 30 commits February 15, 2025 20:58

Converted tests to pytest. Build a Python package. Update requirement…

f4e9cdb

…s.txt and split out requirements-dev.txt. Version bumps.

Restore Python .gitignore

c256244

Extra newline removed

6c3df0b

Merge branch 'master' of github.com:graphframes/graphframes into rjur…

b2838d2

…ney/build-upgrades

Added VERSION file set to 0.8.5

caf5091

isort; fiex edgesDF variable name.

7cfa2d1

Merge branch 'master' of github.com:graphframes/graphframes into rjur…

2ca9a15

…ney/build-upgrades

Back out Dockerfile changes

a8bf0be

Back out version change in build.sbt

54a942d

Backout changes to config and run-tests

8b0e346

Back out pytest conversion

46c2b93

Back out version changes to make nose tests pass

18b5da0

Remove changes to requirements

8eca097

Put nose back in requirements.txt

277c06f

Remove version bump to version.sbt

b55ee48

Remove packages related to testing

f8a8fd9

Remove old setup.py / setup.cfg

bc2cb36

New pyproject.toml and poetry.lock

728be33

Short README for Python package, poetry won't allow a ../README.md path

3cea1a8

Remove requirements files in favor of pyproject.toml

87cc975

Try to poetrize CI build

6f84a5a

pyspark min 3.4

9a8eef0

Local python README in pyproject.toml

75ecd99

Trying to remove he working folder to debug scala issue

80231d0

Set Python working directory again

2a9170b

Accidental newline

3de2263

Install Python for test...

4662717

Run tests from python/ folder

1b7b9f8

Try running tests from python/

58da493

poetry run the unit tests

9f4aa24

Fix Laika docs build: correct #step-6-cleanup anchor and benchmark fa…

3ca78d1

…llback key names

rjurney mentioned this pull request Aug 29, 2026

Create Pregel and Neo4j Ingeration tutorials and add Pregel API to user guide #578

Closed

rjurney requested a lite review from Copilot August 29, 2026 08:16

Copilot started reviewing on behalf of rjurney August 29, 2026 08:16 View session

Copilot AI reviewed Aug 29, 2026

View reviewed changes

Final touches to ship tutorial

fe31d46

james-willis reviewed Aug 31, 2026

View reviewed changes

james-willis mentioned this pull request Aug 31, 2026

ci: run python lint on tests/ and docs/, as its own job #896

Merged

SemyonSinchenko reviewed Sep 1, 2026

View reviewed changes

cursoragent and others added 2 commits September 1, 2026 21:28

rjurney mentioned this pull request Sep 1, 2026

docs(neo4j): use PropertyGraphFrame + validate(), and fix black line-length-88 formatting churn rjurney/graphframes#8

Merged

cursoragent and others added 4 commits September 1, 2026 22:12

Merge pull request #8 from rjurney/cursor/neo4j-property-graphframe-b…

892f335

…lack-fmt-8c8c docs(neo4j): use PropertyGraphFrame + validate(), and fix black line-length-88 formatting churn

Merge branch 'main' into rjurney/pypi-tutorials

37df7b9

rjurney mentioned this pull request Sep 7, 2026

fix(pregel/neo4j): resolve unaddressed PR #809 review feedback rjurney/graphframes#12

Draft

SemyonSinchenko approved these changes Sep 8, 2026

View reviewed changes

SemyonSinchenko requested a review from james-willis September 8, 2026 09:19

rjurney merged commit d989c88 into graphframes:main Sep 8, 2026
12 checks passed

Conversation

rjurney commented Mar 13, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SemyonSinchenko commented Sep 1, 2026

Uh oh!

SemyonSinchenko commented Sep 1, 2026

Uh oh!

SemyonSinchenko commented Sep 1, 2026

Uh oh!

SemyonSinchenko commented Sep 1, 2026

Uh oh!

SemyonSinchenko left a comment

Choose a reason for hiding this comment

Uh oh!

rjurney commented Sep 1, 2026

Uh oh!

rjurney commented Sep 1, 2026

Uh oh!

rjurney commented Sep 1, 2026

Uh oh!

SemyonSinchenko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants