Repository navigation
ci(full-suite): run the performance gate at release checkpoints - #231
Merged
Merged
Conversation
The gate never reaches PASS on the shared dev host (#198). Run it in the full-suite workflow on a dedicated 32-vCPU runner: approved baseline vs this commit, release builds, interleaved on one runner, 20 rounds with one 30-round retry on INCONCLUSIVE. Only PASS passes. A perf_gate_only dispatch input runs the gate alone on any ref.
A/A on the 32-vCPU runner stays INCONCLUSIVE at 30 rounds on 7 required metrics, so failing on INCONCLUSIVE would block every release on runner noise. FAIL and INVALID still fail the checkpoint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Runs the performance gate at release checkpoints, in the full-suite workflow only. Per-PR CI is unchanged. Tracks #198.
What
.github/workflows/full-suite.yml, onblacksmith-32vcpu-ubuntu-2404(the other jobs use 4 vCPU). It compares the approved baseline against the checkpoint commit. Both are built--release --all-featuresand measured interleaved on one runner. The baseline server is cached per commit and toolchain.scripts/perf-gate-ci.shpins the servers to CPUs 2-31, leaving 0-1 for the gate's clients. It runs 20 rounds and, on INCONCLUSIVE, retries once with 30. FAIL or INVALID fails the checkpoint. If the retry is still INCONCLUSIVE, the job passes with a warning annotation, and the summary says plainly that this is not a PASS.performanceand turns boost off where exposed. On these VMs neither is exposed, so the report recordsgovernor unknown.perf-gate-runsartifact holdsrun.json(raw samples),report.mdandverdict.jsonfor 30 days.perf_gate_onlyruns only the gate, on any ref.perf_gate_aameasures the baseline against itself.perf-gate/README.md(new section "At release checkpoints (CI)") andCONTRIBUTING(CI section).Validation (workflow_dispatch on this branch)
Builds take about 3 minutes in total. 20 rounds take about 17.5 minutes, and 30 rounds about 26.
The FAIL is
cheap_query.latency_us.p50: ratio 1.110, interval 1.065..1.196, about 0.115 → 0.126 ms. It is consistent with the sign-off on the dev host (1.048), and the 30-round A/A passed this same metric on this runner. That makes it a plausible real regression of about 10 µs on cheap queries since e732cd5, and it needs investigation before the next checkpoint. Everything else either passes or improved by a large margin: durable inserts got 50x+ faster, and the delta paths and first deltas are faster too.#198
This cannot close yet. The runner is quieter than the dev host, but the A/A run is still INCONCLUSIVE at 30 rounds. The noisy metrics include
insert_single.ack_us.p50(spread up to 325%),insert_batch.ack_us.p50and thedelta_singledelta percentiles. Getting a PASS every time needs a quieter runner, such as bare metal or self-hosted, or a calibrated budget or round decision.