Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
77 changes: 77 additions & 0 deletions .github/workflows/codspeed.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
on:
push:
branches: [main]
pull_request:
types: [unlabeled, opened, synchronize, reopened]
merge_group:
# `workflow_dispatch` allows CodSpeed to trigger backtest
# performance analysis in order to generate initial data.
workflow_dispatch:

name: CodSpeed

permissions:
contents: read

concurrency:
group: ${{ github.workflow }}-${{ github.ref_name }}-${{ github.event.pull_request.number || github.sha }}
cancel-in-progress: true

env:
CARGO_INCREMENTAL: 0
CARGO_TERM_COLOR: always
CI: true

jobs:
benchmarks:
if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip:ci') }}
name: Run benchmarks
runs-on: ubuntu-latest
timeout-minutes: 90
permissions:
contents: read
id-token: write # for OpenID Connect authentication with CodSpeed
env:
CACHE_PATHS: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
target/
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- uses: dtolnay/rust-toolchain@stable

# The benchmarks are linked against libpython through the pyo3 dev-dependency.
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0

- name: Restore cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ env.CACHE_PATHS }}
key: codspeed-${{ hashFiles('**/Cargo.lock') }}
restore-keys: codspeed-
continue-on-error: true

- name: Install cargo-codspeed
run: cargo install cargo-codspeed --locked --version ^5

- name: Build the benchmark targets
run: cargo codspeed build --locked --measurement-mode simulation -p rustpython -p rustpython-sre_engine

- name: Save cache
if: ${{ github.ref == 'refs/heads/main' }} # only save on main
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ${{ env.CACHE_PATHS }}
key: codspeed-${{ hashFiles('**/Cargo.lock') }}
continue-on-error: true

- name: Run the benchmarks
uses: CodSpeedHQ/action@373d6868929f444bc08d901fd0eb0ad52a8875ea # v5.2.1
with:
mode: simulation
run: cargo codspeed run -p rustpython -p rustpython-sre_engine
151 changes: 102 additions & 49 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ cranelift = "0.132.0"
cranelift-jit = "0.132.0"
cranelift-module = "0.132.0"
crc32fast = "1.3.2"
criterion = { version = "0.8", features = ["html_reports"] }
criterion = { package = "codspeed-criterion-compat", version = "5.0.1", features = ["html_reports"] }
crossbeam-utils = "0.8.21"
csv-core = "0.1.11"
digest = "0.11"
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ A Python-3 (CPython >= 3.14.0) Interpreter written in Rust :snake: :scream:

[![Build Status](https://github.com/RustPython/RustPython/workflows/CI/badge.svg)](https://github.com/RustPython/RustPython/actions?query=workflow%3ACI)
[![codecov](https://codecov.io/gh/RustPython/RustPython/branch/main/graph/badge.svg)](https://codecov.io/gh/RustPython/RustPython)
[![CodSpeed](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://app.codspeed.io/RustPython/RustPython?utm_source=badge)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
[![Contributors](https://img.shields.io/github/contributors/RustPython/RustPython.svg)](https://github.com/RustPython/RustPython/graphs/contributors)
[![Discord Shield](https://discordapp.com/api/guilds/1043121930691149845/widget.png?style=shield)][discord]
Expand Down
25 changes: 25 additions & 0 deletions benches/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,31 @@ specific installed Python version by running:
PYTHON_SYS_EXECUTABLE=python3.13 cargo bench
```

## Continuous benchmarking with CodSpeed

The benchmarks are also run on every pull request by
[CodSpeed](https://app.codspeed.io/RustPython/RustPython), which measures them with CPU simulation
instead of wall time. `criterion` is aliased to `codspeed-criterion-compat`, so nothing changes for
`cargo bench`: the compatibility layer only takes over when the CodSpeed runner drives the
benchmarks.

To reproduce a CodSpeed run locally, install [`cargo-codspeed`](https://crates.io/crates/cargo-codspeed)
and the [CodSpeed CLI](https://codspeed.io/docs/cli), then run:

```shell
cargo codspeed build -p rustpython -p rustpython-sre_engine
codspeed run --mode simulation -- cargo codspeed run -p rustpython -p rustpython-sre_engine
```

Two things differ when the benchmarks run under CodSpeed:

- The CPython comparison benchmarks are skipped. They are useful to compare RustPython against
CPython locally, but CodSpeed tracks the evolution of RustPython itself, and running them would
double the duration of an already slow instrumented run.
- The microbenchmarks using `ITERATIONS` run with a single value instead of five. The criterion
benchmark id does not include the iteration count, so all five sizes are reported under the same
name.

### Adding a benchmark

Simply adding a file to the `benchmarks/` directory will add it to the set of files benchmarked. Each file is tested
Expand Down
Loading
Loading