Skip to content

Add CodSpeed continuous benchmarking - #8613

Merged
youknowone merged 1 commit into
mainfrom
codspeed-wizard-1788064449835
Aug 30, 2026
Merged

youknowone merged 1 commit into
mainfrom
codspeed-wizard-1788064449835

Conversation

@codspeed

@codspeed codspeed Bot commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

This wires the existing benchmark suites into CodSpeed so that every pull request gets a performance report, measured with CPU simulation instead of wall time.

Changes

  • Cargo.toml: the workspace criterion dependency is now the codspeed-criterion-compat drop-in (package = "codspeed-criterion-compat"). The compatibility layer only takes over when the CodSpeed runner drives the benchmarks, so cargo bench and cargo criterion keep behaving as before, and no benchmark had to be rewritten. This covers the three existing suites: benches/execution.rs, benches/microbenchmarks.rs and crates/sre_engine/benches/benches.rs.
  • benches/execution.rs, benches/microbenchmarks.rs: two adjustments that only apply when CODSPEED_ENV is set (i.e. under the CodSpeed runner):
    • The CPython reference benchmarks are skipped. They are valuable locally to compare RustPython against CPython, but CodSpeed tracks the evolution of RustPython itself, and measuring CPython would double the duration of an already slow instrumented run.
    • The microbenchmarks using ITERATIONS run with a single size instead of five. The criterion benchmark id does not include the iteration count, so all five sizes are reported under the same name and only the last one would be kept.
  • .github/workflows/codspeed.yaml: new workflow building the benchmark targets with cargo codspeed and running them through CodSpeedHQ/action in simulation mode. Actions are hash-pinned, permissions are granted at the job level, and the cargo/target cache is only saved on main, following the conventions of the existing workflows.
  • README.md / benches/README.md: CodSpeed badge and a short section documenting how to reproduce a CodSpeed run locally.

Verification

The full suite was built and executed locally with the CodSpeed CLI:

cargo codspeed build --measurement-mode simulation -p rustpython -p rustpython-sre_engine
codspeed run --mode simulation -- cargo codspeed run -p rustpython -p rustpython-sre_engine

36 benchmarks are reported (parsing, whole-file execution, pystone, microbenchmarks and the SRE engine patterns) and the instrumented run takes about 95 seconds, so the job is dominated by the build, not by the measurement. cargo check --benches and cargo fmt --check pass, and the new workflow is clean for actionlint and zizmor with the repository configuration.

Notes and possible follow-ups

  • The benchmarks link against libpython through the pyo3 dev-dependency, hence the actions/setup-python step in the workflow.
  • While looking at the results, note that pystone.py and fannkuch.py only measure compilation: their bodies are guarded by if __name__ == '__main__', which is false when the code object is executed in a fresh scope. That is pre-existing behaviour and was left untouched here, but it also means the LOOPS parameter of the pystone group currently has no effect.
  • Once this is merged, CodSpeed will use the main branch as the performance baseline. The first pull requests after the merge will show a comparison as soon as a main run has been recorded.

Alias criterion to codspeed-criterion-compat so the existing benchmark suites can be measured by CodSpeed with CPU simulation on every pull request, and add the workflow running them.
@codspeed

codspeed Bot commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

Congrats! CodSpeed is installed 🎉

🆕 36 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Open in CodSpeed

@codspeed
codspeed Bot marked this pull request as ready for review August 30, 2026 06:04
@codspeed
codspeed Bot requested a review from moreal August 30, 2026 06:06
@youknowone
youknowone merged commit 6af309d into main Aug 30, 2026
32 checks passed
@youknowone
youknowone deleted the codspeed-wizard-1788064449835 branch August 30, 2026 06:18
@youknowone

Copy link
Copy Markdown
Member

Thank you so much!

youknowone pushed a commit that referenced this pull request Sep 16, 2026
Alias criterion to codspeed-criterion-compat so the existing benchmark suites can be measured by CodSpeed with CPU simulation on every pull request, and add the workflow running them.

Co-authored-by: CodSpeed Bot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants