Repository navigation
Extend CI benchmarking with microbenchmarks for individual types - #1087
Conversation
629214a to
2af241e
Compare
Merging this PR will not alter performance
Performance Changes
Comparing 
Footnotes
|
| # re-importing purely within the current interpreter doesn't get reliable timings | ||
| def do_import(): | ||
| subprocess.run([sys.executable, "-c", "import msgspec"], check=True) | ||
| subprocess.run([sys.executable, "-c", f"import {module_name}"], check=True) |
There was a problem hiding this comment.
Can you pass this to hyperfine instead? Otherwise, we really won't get reliable measurements. I'd recommended tracking just the difference/true overhead.
❯ (
uv run --python 3.14 --no-project --isolated --with msgspec
-- hyperfine
--export-json -
--style none
--shell none
--min-runs 100
--warmup 10
--reference-name baseline
--reference 'python -c pass'
--command-name combined
'python -c "import msgspec"'
err> null-device
| from json
| get results
| select command median
| update median {|r| $r.median * 1000 }
| transpose --header-row --as-record
| insert overhead {|r| $r.combined - $r.baseline }
)
╭──────────┬───────╮
│ baseline │ 47.02 │
│ combined │ 87.67 │
│ overhead │ 40.65 │
╰──────────┴───────╯There was a problem hiding this comment.
Not really, no. That will work for local benchmarking, but not in CI with codspeed, where we use its CPU instrumentation, not walltime measurements.
Unfortunately I haven't found a good way yet to measure the import time without the interpreter startup overhead with codspeed.
There was a problem hiding this comment.
Is there no way to submit a raw value that we can track over time?
There was a problem hiding this comment.
Not that I know of, no. I could look into hacking something together, but iiuc that just isn't the intended use case. codspeeds model is tuned towards running stuff you submit there. A bit unfortunate really
There was a problem hiding this comment.
it should be fine as-is though, as we're only tracking the relative performance, not the absolute numbers, and the stability there is exactly what codspeed is attempting to solve with their CPU simulation. i'd suggest we just go with that for a while and see how it works out. with the latest changes i've made, things seem to be quite stable now though
ofek
left a comment
ofek
left a comment
There was a problem hiding this comment.
Thanks!
Can you please open an issue to track startup overhead with hyperfine as conceptualized here using either Codspeed's walltime instrument or some other approach? We really just need any rudimentary way to track the overhead delta from that comment.
@ofek I think we already got #886 tracking this. I'm planning to tackle that soon |
|
Oh that's true, thanks! Sorry to keep talking about startup overhead; it's just that I maintain a great number of CLIs and the pain there is real 😅 |
Before, we only tested overall performance with mixed data. However, this can skew results a bit.
This introduces targeted, isolated tests for builtin types we support, to make it easier to catch regressions / test improvements for individual type handling.