Repository navigation
test(perf-gate): run benchmarks on the benchmark host, add the engine suite, and publish engine baselines - #273
Merged
Merged
Conversation
…s/perf-gate-remote.sh)
…pdates, claims, .why, sessions, memory, recovery, async WAL) and a summary command
…mary.md and --no-verdict
…finity; make perf-gate-remote.sh safe to edit while it runs
…so freed memory cannot hide growth
…r benchmark on the host
…A calibration records; summary skips empty tables
…sion-scale benchmark on the host
…mark that skipped the lock
…chema sessions/v2) A probe whose delta took longer than the 10 s probe timeout counted as missing, and its delta, when it came, as stray, so a saturated server looked like one that loses deltas. Overdue probes now wait for their delta (and 60 s more after the load stops); one that arrives counts as late with its latency.
…00 sessions at 1 and 0.1 writes/s); late, not lost, deltas at saturation
jsam
marked this pull request as ready for review
October 5, 2026 16:06
jsam
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Captain's intent: make the dedicated benchmark the engine performance gate; heavy perf runs go there, the dev box runs only mandatory checks. Scope, as merged into this branch: (1) scripts/perf-gate-remote.sh plus make perf-gate-remote / bench-engine-remote / bench-sessions-remote / PRE_PR_PERF: measures a commit on the host over SSH (fetch or push the commit, build release there, run detached so a dropped connection does not stop it, --attach to collect, copy results back to target/perf-gate/remote/), following the host rules: one benchmark at a time via the shared ~/perf-gate-remote/lock (--wait-lock queues, also waits out foreign inputlayer-server processes), nproc+commit+command recorded in bench.txt, never leave servers running; pins gate clients to CPUs 0-7 and servers to 8-31 (new perf-gate.sh --gate-cpus). (2) Engine suite fixtures in perf-gate (rule_query, unbound_query, writes, claims, why, sessions, memory_facts, memory_graphs, recovery, insert_async) the captain asked for after auditing coverage: measured and reported, deliberately NOT in policy.toml's required set, and the default run set stays the gate's eight fixtures; plus perf-gate summary (absolute numbers), --no-verdict, and the environment recording host CPU count instead of the pinned affinity. (3) bench-sessions (#254's harness) now distinguishes late deltas from missing ones (schema sessions/v2): v1 counted a delta slower than the 10 s probe timeout as missing and its arrival as stray, which made a saturated server look like it loses deltas. (4) Publish the numbers already measured on the host: perf-gate/baselines/engine-baseline-2026-10-04.md (main 25058f1), session-scale-2026-10-05.md (main 8b574ef), A/A calibration records (30-round A/A INCONCLUSIVE on 4/22 required metrics, so the approved baseline e732cd5 is deliberately NOT moved), and a dated 'Engine Baseline' section in BENCHMARKS.md added beside the existing published numbers - the existing 6.83 ms / 11.3 s / 1,652x numbers and their table must not be changed, caveated or removed. Engine only, no end-to-end decision numbers. The benchmark machine is now shut down: do not run benchmarks; validate locally (make perf-gate-check, pre-pr selftest).
What Changed
scripts/perf-gate-remote.shand themake perf-gate-remote/bench-engine-remote/bench-sessions-remotetargets, plusPRE_PR_PERFformake pre-pr. Together they measure a commit on the benchmark hostsam-dev-benchmarksover SSH: the commit is built in release mode on the host and runs detached, so--attachcan collect results after a dropped connection. Results are copied back totarget/perf-gate/remote/<stamp>. Only one benchmark runs at a time, guarded by a shared lock that--wait-lockqueues on. Each run records nproc, commit and command inbench.txtand leaves no server running. A newperf-gate.sh --gate-cpusoption pins gate clients to CPUs 0-7 and servers to 8-31.perf-gate summarycommand for absolute numbers and a--no-verdictflag. The environment record now stores the host CPU count instead of the pinned affinity.sessions/v2, which separates late deltas from missing ones and counts late deltas against the latency budget. The branch also publishes numbers already measured on the host: the engine baseline from 2026-10-04, the session-scale results from 2026-10-05, and A/A calibration reports underperf-gate/baselines/. A dated "Engine Baseline" section is added toBENCHMARKS.mdnext to the existing published numbers, which are unchanged. README, TESTING and Makefile docs are updated to match.Risk Assessment
Testing
Ran perf-gate's unit tests and the pre-pr selftest; both pass. The real remote script and make targets were run against a stand-in benchmark host built locally (ssh/scp shims, throwaway clone, stubbed benchmark scripts, host-side pgrep/pkill limited to harness processes, so another session's real inputlayer-server was not touched). The happy path, exit codes for busy (4), setup error (3), gate FAIL (1) and dropped connection (5), lock queueing, waiting out a foreign server, cleanup of a leftover server, --attach and concurrent runs all behaved as documented. perf-gate summary and compare on the published run files show absolute numbers, exit 3 when a fixture run failed, keep engine fixtures out of the gate, and give the A/A record INCONCLUSIVE. A new focused unit test fails without the late-delta budget rule and passes with it, but that is unit-level coverage, not a live run. Real measurements (pinned gate runs, sessions late-delta drain against a server, recorded host CPU count) were not run because the user forbade benchmarks. This is a CLI/script change with no UI, so there are no screenshots; the evidence is CLI transcripts.
make perf-gate-remoteon a commit origin lacks: it is pushed to the host, built/run detached, and run.json/report.md/verdict.json/summary.md/bench.txt/gate.log are copied to target/pe…make bench-engine-remoteruns--aa --baseline-rev <sha> --fixtures engine --no-verdictpinned;make bench-sessions-remoteruns bench-sessions.sh with servers pinned to 8-31perf-gate summaryprints absolute-number tables from a published run file, and exits 3 listing the failed run when a fixture run erroredEvidence: perf-gate-remote happy path (make perf-gate-remote)
~/.no-mistakes/evidence/01M46D299DA54364PHDNT2KRZK/remote-R1-results)Evidence: Busy host (lock held) exits 4 and still copies gate.log
Evidence: Dirty host clone exits 3
Evidence: Dropped connection then --attach
~/.no-mistakes/evidence/01M46D299DA54364PHDNT2KRZK/remote-host-state)Evidence: perf-gate summary of published engine baseline
Evidence: Engine run judged by policy.toml (engine fixtures under diagnostic)
Evidence: Sessions budget regression test: fails without the fix
Evidence: perf-gate unit tests with the fix
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
perf-gate/src/sessions/mod.rs:324- In sessions/v2, late probes are kept out ofload_delta_ms(lines 518-520). The--max-delta-p99-msbudget check still compares onlyload_delta_ms.p99, andRun::passed()(line 193) ignoreslate_deltas. A probe whose delta took more than 10 s is by definition over any sub-10 s budget, yet it no longer counts against the budget. Example:--sessions 1000 --max-delta-p99-ms 500 --budget-sessions 1000with 1 on-time probe at 300 ms and 5 late probes at 15 s. v1 failed this run (missing deltas). v2 reports p99 = 300 ms and passes with exit 0, so a saturated working tree passes the latency budget. Fix: in the budget loop, fail any candidate run withlate_deltas > 0, or foldlate_delta_msinto the p99 the budget judges. The late count is not split by idle/loaded phase, so counting any late delta against the budget is the simplest sound rule.scripts/perf-gate-remote.sh:210- When the host is busy, run.sh callsfinish 4(lines 129 and 139) before$STAMP.bench.txtexists. On a setup failure it callsfinish 3(lines 149, 152, 154), also before bench.txt is written. The local side then runsscp -q "$HOST:$STATE/$STAMP.bench.txt"unguarded underset -e. scp fails and the script exits 1, which the header documents as a gate FAIL. The documented 'exit 4 when the host is busy' therefore never happens, gate.log (which explains the refusal) is never copied back, and noResults:line is printed. This affectsmake pre-pr PRE_PR_PERF=perf-gate-remoteand--wait-locktimeouts. Fix: make both scp calls tolerant (|| true, or[ -s ]before thesedat line 212), or write a minimal bench.txt before the lock is taken. Then exit with the remote$STATUS.scripts/perf-gate-remote.sh:113- Every invocation rewrites the shared~/perf-gate-remote/run.shin place (cat > $STATE/run.sh, which truncates the same inode). With--wait-lock, a second invocation does this while the first run's bash is still executing that file. Bash reads a script incrementally, so after the long"$SCRIPT"call the first run resumes at its old byte offset in the new content. If the two callers have different versions of perf-gate-remote.sh (for example two worktrees), it runs garbled commands and may never write its status file, so its follower waits forever. If bash reads during the truncate-then-write window, the first run ends early and writes no status. The local script was made edit-safe with{ ... }(be7ec14), but the host-side script was not. Fix: write a per-run copy ($STATE/$STAMP.run.sh) and run that, or wrap the heredoc body in{ ...; exit; }.perf-gate/src/main.rs:69- Simplification:perf-gate summary --arms(arm filtering) is not used by perf-gate.sh, perf-gate-remote.sh, the Makefile or the published baselines, which all summarize every arm pooled. The intent asks only for 'perf-gate summary (absolute numbers)'. Recommend removing the option and theselectedfiltering it drives in summary.rs.perf-gate/src/fixtures/mod.rs:133- Simplification:Fixture::parse_groupadds the groupsgateandallbesidesengine. Onlyengineis needed (make bench-engine-remote). The gate's eight remain the default withoutgate, andallis used by no script or target. The narrower form that meets the intent is theenginegroup alone.🔧 Fix applied.
7 issues (1 error, 3 warnings, 3 infos) still open:
perf-gate/src/sessions/mod.rs:324- In sessions/v2, late probes are kept out ofload_delta_ms(lines 518-520). The--max-delta-p99-msbudget check still compares onlyload_delta_ms.p99, andRun::passed()(line 193) ignoreslate_deltas. A probe whose delta took more than 10 s is by definition over any sub-10 s budget, yet it no longer counts against the budget. Example:--sessions 1000 --max-delta-p99-ms 500 --budget-sessions 1000with 1 on-time probe at 300 ms and 5 late probes at 15 s. v1 failed this run (missing deltas). v2 reports p99 = 300 ms and passes with exit 0, so a saturated working tree passes the latency budget. Fix: in the budget loop, fail any candidate run withlate_deltas > 0, or foldlate_delta_msinto the p99 the budget judges. The late count is not split by idle/loaded phase, so counting any late delta against the budget is the simplest sound rule.scripts/perf-gate-remote.sh:210- When the host is busy, run.sh callsfinish 4(lines 129 and 139) before$STAMP.bench.txtexists. On a setup failure it callsfinish 3(lines 149, 152, 154), also before bench.txt is written. The local side then runsscp -q "$HOST:$STATE/$STAMP.bench.txt"unguarded underset -e. scp fails and the script exits 1, which the header documents as a gate FAIL. The documented 'exit 4 when the host is busy' therefore never happens, gate.log (which explains the refusal) is never copied back, and noResults:line is printed. This affectsmake pre-pr PRE_PR_PERF=perf-gate-remoteand--wait-locktimeouts. Fix: make both scp calls tolerant (|| true, or[ -s ]before thesedat line 212), or write a minimal bench.txt before the lock is taken. Then exit with the remote$STATUS.scripts/perf-gate-remote.sh:113- Every invocation rewrites the shared~/perf-gate-remote/run.shin place (cat > $STATE/run.sh, which truncates the same inode). With--wait-lock, a second invocation does this while the first run's bash is still executing that file. Bash reads a script incrementally, so after the long"$SCRIPT"call the first run resumes at its old byte offset in the new content. If the two callers have different versions of perf-gate-remote.sh (for example two worktrees), it runs garbled commands and may never write its status file, so its follower waits forever. If bash reads during the truncate-then-write window, the first run ends early and writes no status. The local script was made edit-safe with{ ... }(be7ec14), but the host-side script was not. Fix: write a per-run copy ($STATE/$STAMP.run.sh) and run that, or wrap the heredoc body in{ ...; exit; }.perf-gate/src/main.rs:69- Simplification:perf-gate summary --arms(arm filtering) is not used by perf-gate.sh, perf-gate-remote.sh, the Makefile or the published baselines, which all summarize every arm pooled. The intent asks only for 'perf-gate summary (absolute numbers)'. Recommend removing the option and theselectedfiltering it drives in summary.rs.perf-gate/src/fixtures/mod.rs:133- Simplification:Fixture::parse_groupadds the groupsgateandallbesidesengine. Onlyengineis needed (make bench-engine-remote). The gate's eight remain the default withoutgate, andallis used by no script or target. The narrower form that meets the intent is theenginegroup alone.perf-gate/target/debug/perf-gate:1- The round-1 fix commit 3a03e83 also committed perf-gate's Cargo build directory: 1,138 files under perf-gate/target/, including an 81 MB debug binary (perf-gate/target/debug/perf-gate), .cargo-lock files, fingerprints and .rustc_info.json. The fix round introduced this. ab7ebf7 tracked none of these files. The cause is that the root .gitignore only ignores/target/, which is anchored to the repo root, and perf-gate is a standalone workspace with its own target dir, so building it locally leaves untracked artifacts thatgit add -Apicks up. Merging this would add host-specific binaries and about 80 MB to history. Fix:git rm -r --cached perf-gate/targetand amend or follow up so no tracked file remains under perf-gate/target/. To stop it happening again, ignoreperf-gate/target/(or change/target/totarget/) in .gitignore.scripts/perf-gate-remote.sh:113- Round-1 fix: each run now writes~/perf-gate-remote/<stamp>.run.shand never deletes it. The old shared run.sh stays on the host too. These are small files that pile up next to the per-stamp logs and status files, which are already kept, so this is consistent with the existing retention. Noted only.🔧 Fix applied.
4 issues (1 error, 1 warning, 2 infos) still open:
perf-gate/src/main.rs:69- Simplification:perf-gate summary --arms(arm filtering) is not used by perf-gate.sh, perf-gate-remote.sh, the Makefile or the published baselines, which all summarize every arm pooled. The intent asks only for 'perf-gate summary (absolute numbers)'. Recommend removing the option and theselectedfiltering it drives in summary.rs.perf-gate/src/fixtures/mod.rs:133- Simplification:Fixture::parse_groupadds the groupsgateandallbesidesengine. Onlyengineis needed (make bench-engine-remote). The gate's eight remain the default withoutgate, andallis used by no script or target. The narrower form that meets the intent is theenginegroup alone.perf-gate/target/debug/perf-gate:1- The round-1 fix commit 3a03e83 also committed perf-gate's Cargo build directory: 1,138 files under perf-gate/target/, including an 81 MB debug binary (perf-gate/target/debug/perf-gate), .cargo-lock files, fingerprints and .rustc_info.json. The fix round introduced this. ab7ebf7 tracked none of these files. The cause is that the root .gitignore only ignores/target/, which is anchored to the repo root, and perf-gate is a standalone workspace with its own target dir, so building it locally leaves untracked artifacts thatgit add -Apicks up. Merging this would add host-specific binaries and about 80 MB to history. Fix:git rm -r --cached perf-gate/targetand amend or follow up so no tracked file remains under perf-gate/target/. To stop it happening again, ignoreperf-gate/target/(or change/target/totarget/) in .gitignore.scripts/perf-gate-remote.sh:113- Round-1 fix: each run now writes~/perf-gate-remote/<stamp>.run.shand never deletes it. The old shared run.sh stays on the host too. These are small files that pile up next to the per-stamp logs and status files, which are already kept, so this is consistent with the existing retention. Noted only.✅ **Test** - passed
✅ No issues found.
make perf-gate-remoteon a commit origin lacks: it is pushed to the host, built/run detached, and run.json/report.md/verdict.json/summary.md/bench.txt/gate.log are copied to target/pe…make bench-engine-remoteruns--aa --baseline-rev <sha> --fixtures engine --no-verdictpinned;make bench-sessions-remoteruns bench-sessions.sh with servers pinned to 8-31perf-gate summaryprints absolute-number tables from a published run file, and exits 3 listing the failed run when a fixture run erroredcargo test --manifest-path perf-gate/Cargo.toml --target-dir target(the test half ofmake perf-gate-check; clippy not run per the no-linters rule): 59/59 after adding the sessions testspython3 -m unittest scripts/test_pre_pr.py(the pre-pr selftest): 5 OKperf-gate summaryon the published engine and memory run.json.gz files, plus a copy with an injected fixture error (exit 3)perf-gate compare --policy perf-gate/policy.tomlon the engine-baseline run (engine fixtures listed under diagnostic, not gated) and on the 25058f16 A/A record (INCONCLUSIVE, 4/22)Stand-in host: ssh/scp shims running commands against a throwaway HOME holding a git clone, with host-side pgrep/pkill limited to processes tagged by the harness. A throwaway local clone with a stub-benchmark commit drove the realscripts/perf-gate-remote.shand the Makefilemake perf-gate-remote PERF_GATE_HOST=fakehost(commit missing from origin gets pushed, detached run, files copied back, bench.txt, CPU pinning)perf-gate-remote: gate exit 1 passed through; lock held gives exit 4;--wait-lock 60queues behind the lock; foreign server gives exit 4;--wait-lockwaits the server out; dirty host clone gives exit 3; a server the benchmark leaves behind is killedDropped connection (killed the follower over ssh) exits 5 with an --attach hint while the host run continues;--attach STAMPcollects the finished runTwo concurrent invocations from two different versions of the script, with--wait-lock 120: they run one after the other, each with its own <stamp>.run.sh, both exit 0make bench-engine-remoteandmake bench-sessions-remote SESSIONS_ARGS=...: commands recorded in bench.txtNew tests inperf-gate/src/sessions/mod.rs(sessions::tests::*): the budget test fails with the late check disabled and passes with itgit diff 22ceb6b6..HEAD -- BENCHMARKS.md: 0 lines removed; 6.83 ms / 11.3 s / 1,652x still present; 'Engine Baseline (2026-10-04)' section added✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.