Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: graphframes/graphframes
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: main
Choose a base ref
...
head repository: graphframes/graphframes
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: master
Choose a head ref
Checking mergeability… Don’t worry, you can still create the pull request.
  • 1 commit
  • 13 files changed
  • 1 contributor

Commits on Sep 2, 2026

  1. feat: set Spark job descriptions for iterative algorithms (#898) 

    * feat: set Spark job descriptions for iterative algorithms

Set per-iteration Spark job descriptions (shown in the Jobs/Stages pages
of the Spark UI) for Pregel and both DataFrame-based connected
components implementations, so the progress of long runs is visible
without reading driver logs. Pregel-based algorithms report their own
algorithm name, and Pregel exposes setJobDescriptionPrefix to tell
concurrent runs apart. The caller's job description is restored after
the run, including on failure.

Closes #797

* feat: cover random walks and MIS with job descriptions

RandomWalkBase sets per-batch job descriptions, with the prefix
overridable per subclass (RandomWalkWithRestart reports its own name),
and MaximalIndependentSet labels its iterations. Requested in review.
    @james-willis
    james-willis authored Sep 2, 2026
    Configuration menu
    Copy the full SHA
    41a804d View commit details
    Browse the repository at this point in the history
Loading