Restore pytest-benchmark for local and manual profiling - #1113

@ofek good call, reworked to your suggestion. Instead of dropping the pytest-benchmark scaffolding, this just re-adds pytest-benchmark to test-prof. It was removed in #1075, which is what orphaned the --benchmark-* flags and crashed the workflow.

That restores the simple benchmarks for local runs and the manual profile trigger. I confirmed the two plugins coexist: without --codspeed, pytest-benchmark runs the benchmark fixture (walltime table); with --codspeed, codspeed instruments it and pytest-benchmark steps aside.

One thing still open from the old setup: the --calibrate/--rounds conftest options are accepted but no longer wired to anything (the bench/bench_config fixtures that consumed them were also removed in #1075, the tests now call benchmark.pedantic(...) directly). This PR doesn't touch that. If you want those knobs functional again it's a bit more surgery on the tests, happy to leave it to you since you mentioned wanting to take this, or fold it in here. Your call.

No rush on my end, just let me know what you think whenever you get a chance.

Merging this PR will not alter performance

✅ 139 untouched benchmarks
⏩ 135 skipped benchmarks¹

_{Comparing fix-profile-workflow-benchmark-flags (81cf3aa) with main (3f67b34)}

I'll take on the work to bring them back, thanks!

Whatever's easiest for you: feel free to merge this and start on yours, or close it and just repeat the one line in your own PR since it's the only change here. I'll leave the fate of this one in your hands.

Merged current main in. Still the same one line: pytest-benchmark back in the test-prof group.

@ofek two months on, pytest-benchmark is still absent from the repo and .justfile:136 still appends --benchmark-name short --benchmark-disable-gc, so just test-perf <name> and profile.yml (just test-perf all --calibrate --rounds 10000) still die at collection. The last three Manual Profile runs, all on 2026-06-20, failed for exactly this; the last successful one was 2025-11-16.

Your changes-requested was against the earlier version of this PR, which removed the scaffolding instead. This one is the minimal form of the restoration you said you would take on, and it does not conflict with a larger one later.

Checked that it is safe for CodSpeed: codspeed.yml syncs the same group, and pytest-codspeed archives a foreign benchmark fixture and substitutes its own, so the benchmark counts are unchanged.

If you want to own the whole thing, say so and I will close this. Otherwise I plan to merge it in a few days as a CI fix.

@ofek @provinzkraut I have merged main into this branch. The change now consists only of restoring the missing pytest-benchmark dependency; the profiling infrastructure is retained.

Ofek, your changes-requested review concerned the earlier version that removed that infrastructure. Could you re-review the current version? Janek, could you review compatibility with CodSpeed?

@Siyet I think we can just revert the whole codspeed stuff. It does not work as promised for us, and I see no value in keeping it

CodSpeed workflow runs are now disabled for the whole repository. CI is green on this PR; merging is currently blocked only by ofek's earlier changes-requested review. @ofek pls take another look at the current version.