Skip to content

fix(perf-gate): build every baseline in a target directory of its own - #406

Closed
jsam wants to merge 4 commits into
mainfrom
fm/il-perf-gate-stale-baseline
Closed

fix(perf-gate): build every baseline in a target directory of its own#406
jsam wants to merge 4 commits into
mainfrom
fm/il-perf-gate-stale-baseline

Conversation

@jsam

@jsam jsam commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Intent

Performance regressions and the perf gate in inputlayer/inputlayer are part of getting the engine production ready, and benchmarks run only on the benchmark host. The perf gate there currently produces wrong or failed baseline builds, found while bisecting #374:

scripts/perf-gate.sh extracts each baseline with git archive | tar -x (file mtimes = the commit's time) and builds every baseline in one shared target/perf-gate/build. Cargo's metadata hash does not include the path of a path dependency, so every baseline's inputlayer-ws-protocol maps to the same artifact, and an older baseline extracted after a newer one was built sees files older than the fingerprint and reuses the newer crate. The build fails when the APIs diverged (missing field code in ServerFrame, from #387 built by another lane's run on the shared host clone), and silently links the wrong crate when they did not. Five bisect steps failed this way. Fix: touch the extracted tree, or a per-baseline CARGO_TARGET_DIR. Until then only baselines at or after the newest-built ws-protocol build.

Fix it so every baseline is built from exactly its own sources. Deliver it as a small PR; done means merged with CI green.

What Changed

  • New scripts/perf-gate-baseline.sh builds one baseline commit from git archive with its own CARGO_TARGET_DIR (target/perf-gate/baseline-build/<commit>-<toolchain>/) and prints the path of the cached binary. The build directory is removed when the build ends, including on failure and on SIGTERM/SIGHUP. scripts/perf-gate.sh, scripts/bench-views.sh and scripts/bench-sessions.sh now call it instead of each carrying a copy of the build that shared target/perf-gate/build.
  • Cached baseline binaries move from target/perf-gate/servers/ to target/perf-gate/baseline-servers/, so binaries built in the shared directory are not reused. The CI cache path and key in full-suite.yml change to match.
  • New scripts/perf-gate-baseline-selftest.sh builds two commits of a throwaway workspace with a path dependency, in both orders, and checks that each binary comes from its own commit, that a cached baseline is not rebuilt, and that a failed or killed build leaves no build directory. It runs in make pre-pr-selftest and in the perf-gate-tool CI job; test_pre_pr.py, perf-gate/README.md, CONTRIBUTING and TESTING.md are updated to match.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change replaces three identical inline baseline builds with one small helper that gives each baseline its own source and target directory and a fresh cache location, so no stale or cross-commit artifact is reachable, and the earlier fix round's EXIT trap and plain cp read correctly; nothing was executed in this review.

Testing

No baseline commands had run before this step. I ran the baseline selftest on the target, then against the base commit's shared-directory logic and against the helper without this round's signal trap; both of those fail on exactly the checks they should. On the real repository I built #387 (efa32ce) and then its parent (527bacc) with the new helper and with the base logic: the new helper gives two different servers, the base logic gives the parent #387's binary. I interrupted the helper mid-build with SIGINT, SIGTERM and SIGHUP, to the process group and to the helper alone, on a loaded host; no build directory was left in any try. The perf gate (quick profile, which reports INVALID and exits 3 by design), the session benchmark and the views benchmark each ran a baseline through the helper from the cache without rebuilding it. The pre-pr routing tests pass. Build outputs are removed and the worktree is clean. Nothing here is UI-facing, so the evidence is CLI transcripts.

  • Live validation: ✅ go - 8 of 8 scenarios driven live against the product
Scenario Result Live Evidence
Build the baseline of #387 (efa32ce) and then of its older parent (527bacc) on the real repository: each server is its own commit's ✅ pass live round3/real-new.txt: both exit 0, each build compiles inputlayer-ws-protocol itself; efa32ce sha256 cc89631e with 10 access_denied strings, 527bacc sha256 4382b65d with 0; baseline-build empty aft…
The same two builds with the base commit's shared build directory reproduce the reported fault: the older baseline is the newer commit's server ✅ pass live round3/real-old.txt: the 527bacc build finishes in 0.25s without compiling ws-protocol, and its binary is byte-identical to efa32ce's (sha256 cc89631e, 10 access_denied strings)
Fixture workspace, newer-then-older and older-then-newer: each baseline server prints its own commit's engine and protocol; the base logic does not ✅ pass live round3/selftest-target.log (all PASS, exit 0) against round3/selftest-base-logic.log (2 FAIL: 'engine-newer protocol-newer' for the older baseline and the reverse, exit 1)
Request an already built baseline again: it is returned from the cache without a rebuild ✅ pass live round3/selftest-target.log 'a cached baseline is not built again'; round3/gate-quick.log, bench-sessions.log and bench-views.log contain no 'Build baseline server' for the cached 527bacc
Request a baseline that does not compile: the helper fails and leaves no build directory behind ✅ pass live round3/selftest-target.log: 'a baseline that does not compile fails' and 'no build directory is left behind, built or not'
Kill a baseline build with SIGINT, SIGTERM or SIGHUP, to the process group or to the helper alone: no build directory is left behind and no partial binary is cached ✅ pass live round3/interrupt-group.log: 9 of 9 clean under load average about 30; round3/interrupt-group-without-signal-trap.log: the helper without the trap leaves the directory in 4 of 6 SIGTERM/SIGHUP tries; r…
Run the perf gate against a baseline commit: it gets the baseline from the helper and measures both servers ✅ pass live round3/gate-quick.log: baseline 527bacc against the efa32ce server, 8 fixture runs over 2 rounds, report written; exit 3 with 'Performance gate: INVALID' is what the quick profile always gives
Run the session and views benchmarks with --baseline-rev: both measure the helper's baseline binary beside the working tree's ✅ pass live round3/bench-sessions.log and round3/bench-views.log: baseline path is target/perf-gate/baseline-servers/527bacc3...-8dc912acd5bf/inputlayer-server, both exit 0
Evidence: Real repository, new helper: #387 then its parent

$ new efa32cec -> exit=0 seconds=185; cargo: Finished release profile [optimized] target(s) in 3m 04s ws-protocol compiled in this build: 1 baseline-servers/efa32cec62979f089624c76f6846c9b9e27b4381-8dc912acd5bf/inputlayer-server sha256=cc89631edeaa6a8b access_denied strings=10 $ new 527bacc3 -> exit=0 seconds=175; cargo: Finished release profile [optimized] target(s) in 2m 53s ws-protocol compiled in this build: 1 baseline-servers/527bacc306ae480504adbdec56c6823672629793-8dc912acd5bf/inputlayer-server sha256=4382b65d6c577d86 access_denied strings=0 baseline-build afterwards: []

$ new efa32cec -> exit=0 seconds=185; cargo:     Finished `release` profile [optimized] target(s) in 3m 04s 
  ws-protocol compiled in this build: 1
  baseline-servers/efa32cec62979f089624c76f6846c9b9e27b4381-8dc912acd5bf/inputlayer-server  sha256=cc89631edeaa6a8b  access_denied strings=10
$ new 527bacc3 -> exit=0 seconds=175; cargo:     Finished `release` profile [optimized] target(s) in 2m 53s 
  ws-protocol compiled in this build: 1
  baseline-servers/527bacc306ae480504adbdec56c6823672629793-8dc912acd5bf/inputlayer-server  sha256=4382b65d6c577d86  access_denied strings=0
baseline-build afterwards: []

real	6m0.898s
user	33m0.587s
sys	2m41.830s
Evidence: Real repository, base-commit logic: the parent gets #387's server

$ old efa32cec -> exit=0 seconds=186; cargo: Finished release profile [optimized] target(s) in 3m 06s ws-protocol compiled in this build: 1 servers/efa32cec62979f089624c76f6846c9b9e27b4381-415a6fa19307/inputlayer-server sha256=cc89631edeaa6a8b access_denied strings=10 $ old 527bacc3 -> exit=0 seconds=1; cargo: Finished release profile [optimized] target(s) in 0.25s ws-protocol compiled in this build: 0 servers/527bacc306ae480504adbdec56c6823672629793-415a6fa19307/inputlayer-server sha256=cc89631edeaa6a8b access_denied strings=10

$ old efa32cec -> exit=0 seconds=186; cargo:     Finished `release` profile [optimized] target(s) in 3m 06s 
  ws-protocol compiled in this build: 1
  servers/efa32cec62979f089624c76f6846c9b9e27b4381-415a6fa19307/inputlayer-server  sha256=cc89631edeaa6a8b  access_denied strings=10
$ old 527bacc3 -> exit=0 seconds=1; cargo:     Finished `release` profile [optimized] target(s) in 0.25s 
  ws-protocol compiled in this build: 0
  servers/527bacc306ae480504adbdec56c6823672629793-415a6fa19307/inputlayer-server  sha256=cc89631edeaa6a8b  access_denied strings=10
baseline-build afterwards: [527bacc306ae480504adbdec56c6823672629793-8dc912acd5bf]

real	3m7.399s
user	16m27.996s
sys	1m21.610s
Evidence: Baseline selftest at 1de89a4

PASS: the newer baseline is built from its own sources PASS: the older baseline is built from its own sources PASS: the older baseline is built from its own sources PASS: the newer baseline is built from its own sources PASS: a cached baseline is not built again PASS: the older baseline is built from its own sources PASS: a baseline that does not compile fails PASS: no build directory is left behind, built or not PASS: a build killed with SIGTERM leaves no build directory behind PASS: a build killed with SIGHUP leaves no build directory behind perf-gate-baseline-selftest: all checks passed exit=0

PASS: the newer baseline is built from its own sources
PASS: the older baseline is built from its own sources
PASS: the older baseline is built from its own sources
PASS: the newer baseline is built from its own sources
PASS: a cached baseline is not built again
PASS: the older baseline is built from its own sources
PASS: a baseline that does not compile fails
PASS: no build directory is left behind, built or not
PASS: a build killed with SIGTERM leaves no build directory behind
PASS: a build killed with SIGHUP leaves no build directory behind
perf-gate-baseline-selftest: all checks passed
exit=0
Evidence: Same selftest against the base commit's shared build directory

PASS: the newer baseline is built from its own sources FAIL: the older baseline was built from other sources: it says 'engine-newer protocol-newer' PASS: the older baseline is built from its own sources FAIL: the newer baseline was built from other sources: it says 'engine-older protocol-older' ... perf-gate-baseline-selftest: 2 failure(s) exit=1

PASS: the newer baseline is built from its own sources
FAIL: the older baseline was built from other sources: it says 'engine-newer protocol-newer'
PASS: the older baseline is built from its own sources
FAIL: the newer baseline was built from other sources: it says 'engine-older protocol-older'
PASS: a cached baseline is not built again
PASS: the older baseline is built from its own sources
PASS: a baseline that does not compile fails
ls: cannot access '/tmp/tmp.njRRpv8Ikv/older-first/target/perf-gate/baseline-build': No such file or directory
PASS: no build directory is left behind, built or not
perf-gate-baseline-selftest: 2 failure(s)
exit=1
Evidence: Selftest kill checks against the helper without the TERM/HUP trap

FAIL: a build killed with SIGTERM left its build directory behind FAIL: a build killed with SIGHUP left its build directory behind perf-gate-baseline-selftest: 2 failure(s) exit=1

FAIL: a build killed with SIGTERM left its build directory behind
FAIL: a build killed with SIGHUP left its build directory behind
perf-gate-baseline-selftest: 2 failure(s)
exit=1
Evidence: Process-group interrupts mid-build, helper at 1de89a4

SIGINT #1..#3: helper exit 124; baseline-build holds [] right after; [] 2s later SIGTERM #1..#3: helper exit 124; baseline-build holds [] right after; [] 2s later SIGHUP #1..#3: helper exit 124; baseline-build holds [] right after; [] 2s later

# scripts/perf-gate-baseline.sh at 1de89a44, signal to the helper's whole process group 8s into a 40s build, 3 tries per signal; uptime:  06:37:38 up 10:27,  3 users,  load average: 30.47, 34.01, 37.19
SIGINT #1: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGINT #2: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGINT #3: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGTERM #1: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGTERM #2: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGTERM #3: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGHUP #1: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGHUP #2: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGHUP #3: helper exit 124; baseline-build holds [] right after; [] 2s later
Evidence: Process-group interrupts, helper without the TERM/HUP trap (4 of 6 SIGTERM/SIGHUP tries leave the directory) 
# the same helper without the TERM/HUP trap line (as at 2280c37c), same load; uptime:  06:39:09 up 10:29,  3 users,  load average: 28.66, 33.44, 36.75
SIGINT #1: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGINT #2: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGINT #3: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGTERM #1: helper exit 124; baseline-build holds [4ac83f98ffe40890346454762fbe06ad474f1590-8dc912acd5bf] right after; [4ac83f98ffe40890346454762fbe06ad474f1590-8dc912acd5bf] 2s later
SIGTERM #2: helper exit 124; baseline-build holds [4ac83f98ffe40890346454762fbe06ad474f1590-8dc912acd5bf] right after; [4ac83f98ffe40890346454762fbe06ad474f1590-8dc912acd5bf] 2s later
SIGTERM #3: helper exit 124; baseline-build holds [4ac83f98ffe40890346454762fbe06ad474f1590-8dc912acd5bf] right after; [4ac83f98ffe40890346454762fbe06ad474f1590-8dc912acd5bf] 2s later
SIGHUP #1: helper exit 124; baseline-build holds [4ac83f98ffe40890346454762fbe06ad474f1590-8dc912acd5bf] right after; [4ac83f98ffe40890346454762fbe06ad474f1590-8dc912acd5bf] 2s later
SIGHUP #2: helper exit 124; baseline-build holds [] right after; [] 2s later
SIGHUP #3: helper exit 124; baseline-build holds [] right after; [] 2s later
Evidence: Signals to the helper process alone

SIGTERM to the helper alone at 5s: helper exit 143 after 22s; baseline-build []; baseline-servers []; cargo still running: 0 SIGHUP to the helper alone at 5s: helper exit 143 after 20s; baseline-build []; baseline-servers []; cargo still running: 0 SIGINT to the helper alone at 5s: helper exit 0 after 20s; baseline-build []; baseline-servers [0a561187...-8dc912acd5bf]; cargo still running: 0

SIGTERM to the helper alone at 5s: helper exit 143 after 22s; baseline-build []; baseline-servers []; cargo still running: 0
  22s later: baseline-build []; baseline-servers []
SIGHUP to the helper alone at 5s: helper exit 143 after 20s; baseline-build []; baseline-servers []; cargo still running: 0
  22s later: baseline-build []; baseline-servers []
SIGINT to the helper alone at 5s: helper exit 0 after 20s; baseline-build []; baseline-servers [0a5611875bbf7277a76bbe9a6637ed39ce576402-8dc912acd5bf]; cargo still running: 0
  22s later: baseline-build []; baseline-servers [0a5611875bbf7277a76bbe9a6637ed39ce576402-8dc912acd5bf]
Evidence: Perf gate, quick profile, baseline 527bacc from the helper 
++ scripts/perf-gate.sh --baseline-rev 527bacc3 --candidate-bin target/perf-gate/baseline-servers/efa32cec62979f089624c76f6846c9b9e27b4381-8dc912acd5bf/inputlayer-server --profile quick --rounds 2 --fixtures cheap_query,insert_single
=== Build perf-gate ===
=== Measure (baseline 527bacc306ae480504adbdec56c6823672629793 vs candidate binary ~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/baseline-servers/efa32cec62979f089624c76f6846c9b9e27b4381-8dc912acd5bf/inputlayer-server) ===
round 1/2 cheap_query    baseline  concurrent_latency_us p50 0.89ms p99 2.29ms | latency_us p50 0.51ms p99 1.32ms | queries_per_sec 8117
round 1/2 cheap_query    candidate concurrent_latency_us p50 0.98ms p99 2.78ms | latency_us p50 0.47ms p99 1.22ms | queries_per_sec 7025
round 1/2 insert_single  candidate ack_us p50 38.11ms p99 286.97ms | concurrent_ack_us p50 660.67ms p99 972.36ms | concurrent_facts_per_sec 6 | facts_per_sec 11
round 1/2 insert_single  baseline  ack_us p50 177.20ms p99 820.45ms | concurrent_ack_us p50 32.07ms p99 1066.51ms | concurrent_facts_per_sec 20 | facts_per_sec 5
round 2/2 cheap_query    candidate concurrent_latency_us p50 0.89ms p99 2.97ms | latency_us p50 0.52ms p99 3.80ms | queries_per_sec 7495
round 2/2 cheap_query    baseline  concurrent_latency_us p50 0.96ms p99 3.11ms | latency_us p50 0.53ms p99 1.12ms | queries_per_sec 6764
round 2/2 insert_single  baseline  ack_us p50 1.64ms p99 10.17ms | concurrent_ack_us p50 6.52ms p99 12.44ms | concurrent_facts_per_sec 551 | facts_per_sec 499
round 2/2 insert_single  candidate ack_us p50 1.81ms p99 10.93ms | concurrent_ack_us p50 7.67ms p99 16.78ms | concurrent_facts_per_sec 442 | facts_per_sec 445
run file: ~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/runs/20261006T064113Z/run.json
=== Judge ===
## Performance gate: INVALID

Policy **provisional**: cost budget p50 +5%, p99 +10%, throughput -5%; 95% interval of the median paired per-round ratio. Budgets are the plan's proposed ceilings, pending captain decision D1; tightening is always allowed.

- baseline: `527bacc306ae480504adbdec56c6823672629793` (binary sha256 4382b65d6c57)
- candidate: `binary ~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/baseline-servers/efa32cec62979f089624c76f6846c9b9e27b4381-8dc912acd5bf/inputlayer-server` (binary sha256 cc89631edeaa)
- profile `quick`, 2 rounds, fixtures cheap_query, insert_single
- host `computeinstance-e00w59r9a240k9qz1e`: AMD EPYC-Genoa Processor (32 CPUs, 125 GB), kernel 6.11.0-1016-nvidia, governor unknown, data on ext4 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/runs/20261006T064113Z/servers), server CPUs unpinned
- load average: start `31.13 33.76 36.52 22/1530 1676222`, end `27.31 32.59 36.04 9/1556 1688839`
- **busy host**: 1-minute load 31 on 32 CPUs; results carry other work's noise
- build: rustc 1.99.0 (b940084d7 2026-09-28); cargo build --release --all-features

\### Problems

- bound_query.concurrent_latency_us.p99: baseline: 0 valid rounds, need 6
- bound_query.latency_us.p50: baseline: 0 valid rounds, need 6
- bound_query.latency_us.p99: baseline: 0 valid rounds, need 6
- bound_query.queries_per_sec: baseline: 0 valid rounds, need 6
- cheap_query.latency_us.p50: baseline: 2 valid rounds, need 6
- cheap_query.latency_us.p99: baseline round 1: 80 samples, need 100
- cheap_query.queries_per_sec: baseline: 2 valid rounds, need 6
- delta_fanout.delta_us.p50: baseline: 0 valid rounds, need 6
- delta_fanout.delta_us.p99: baseline: 0 valid rounds, need 6
- delta_fanout.last_agent_us.p99: baseline: 0 valid rounds, need 6
- delta_first.first_delta_us.p50: baseline: 0 valid rounds, need 6
- delta_first.warm_delta_us.p50: baseline: 0 valid rounds, need 6
- delta_keyed.ack_us.p99: baseline: 0 valid rounds, need 6
- delta_keyed.delta_us.p50: baseline: 0 valid rounds, need 6
- delta_keyed.delta_us.p99: baseline: 0 valid rounds, need 6
- delta_keyed.writes_per_server_cpu_sec: baseline: 0 valid rounds, need 6
- delta_single.delta_us.p50: baseline: 0 valid rounds, need 6
- delta_single.delta_us.p99: baseline: 0 valid rounds, need 6
- insert_batch.ack_us.p50: baseline: 0 valid rounds, need 6
- insert_batch.facts_per_sec: baseline: 0 valid rounds, need 6
- insert_single.ack_us.p50: baseline: 2 valid rounds, need 6
- insert_single.ack_us.p99: baseline round 1: 40 samples, need 100
- insert_single.concurrent_facts_per_sec: baseline: 2 valid rounds, need 6
- insert_single.facts_per_sec: baseline: 2 valid rounds, need 6
- interference.delta_us.p50: baseline: 0 valid rounds, need 6
- interference.delta_us.p99: baseline: 0 valid rounds, need 6

\### Required metrics

| metric | baseline | candidate | cost ratio | interval | budget | baseline spread | status |
|---|---|---|---|---|---|---|---|
| bound_query.concurrent_latency_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline: 0 valid rounds, need 6 |
| bound_query.latency_us.p50 | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| bound_query.latency_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline: 0 valid rounds, need 6 |
| bound_query.queries_per_sec | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| cheap_query.latency_us.p50 | 0.516 ms | 0.493 ms | 0.954 | - | 1.05 | 4.5% | INVALID: baseline: 2 valid rounds, need 6 |
| cheap_query.latency_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline round 1: 80 samples, need 100 |
| cheap_query.queries_per_sec | 7440/s | 7260/s | 1.029 | - | 1.05 | 18.2% | INVALID: baseline: 2 valid rounds, need 6 |
| delta_fanout.delta_us.p50 | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| delta_fanout.delta_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline: 0 valid rounds, need 6 |
| delta_fanout.last_agent_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline: 0 valid rounds, need 6 |
| delta_first.first_delta_us.p50 | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| delta_first.warm_delta_us.p50 | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| delta_keyed.ack_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline: 0 valid rounds, need 6 |
| delta_keyed.delta_us.p50 | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| delta_keyed.delta_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline: 0 valid rounds, need 6 |
| delta_keyed.writes_per_server_cpu_sec | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| delta_single.delta_us.p50 | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| delta_single.delta_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline: 0 valid rounds, need 6 |
| insert_batch.ack_us.p50 | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| insert_batch.facts_per_sec | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| insert_single.ack_us.p50 | 89.421 ms | 19.959 ms | 0.657 | - | 1.05 | 196.3% | INVALID: baseline: 2 valid rounds, need 6 |
| insert_single.ack_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline round 1: 40 samples, need 100 |
| insert_single.concurrent_facts_per_sec | 286/s | 224/s | 2.357 | - | 1.05 | 185.7% | INVALID: baseline: 2 valid rounds, need 6 |
| insert_single.facts_per_sec | 252/s | 228/s | 0.774 | - | 1.05 | 196.2% | INVALID: baseline: 2 valid rounds, need 6 |
| interference.delta_us.p50 | - | - | - | - | 1.05 | - | INVALID: baseline: 0 valid rounds, need 6 |
| interference.delta_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline: 0 valid rounds, need 6 |

\### Diagnostic metrics (not gated)

| metric | baseline | candidate | cost ratio | interval | budget | baseline spread | status |
|---|---|---|---|---|---|---|---|
| cheap_query.concurrent_latency_us.p50 | 0.925 ms | 0.933 ms | 1.012 | - | 1.05 | 7.8% | INVALID: baseline: 2 valid rounds, need 6 |
| cheap_query.concurrent_latency_us.p99 | 2.696 ms | 2.877 ms | 1.086 | - | 1.10 | 30.4% | INVALID: baseline: 2 valid rounds, need 6 |
| insert_single.concurrent_ack_us.p50 | 19.297 ms | 334.168 ms | 10.887 | - | 1.05 | 132.4% | INVALID: baseline: 2 valid rounds, need 6 |
| insert_single.concurrent_ack_us.p99 | - | - | - | - | 1.10 | - | INVALID: baseline round 1: 40 samples, need 100 |

Report: ~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/runs/20261006T064113Z/report.md (attach to the PR record)
++ echo exit=3
exit=3
Evidence: Session benchmark with --baseline-rev 527bacc 
++ scripts/bench-sessions.sh --baseline-rev 527bacc3 --sessions 1,5 --load-secs 2
=== Build perf-gate and server (release, all features) ===
   Compiling cfg-if v1.0.5
   Compiling libc v0.2.190
   Compiling serde_core v1.0.229
   Compiling bytes v1.12.1
   Compiling once_cell v1.21.4
   Compiling autocfg v1.5.1
   Compiling libm v0.2.16
   Compiling zerocopy v0.8.60
   Compiling memchr v2.8.3
   Compiling itoa v1.0.18
   Compiling equivalent v1.0.2
   Compiling log v0.4.34
   Compiling find-msvc-tools v0.1.14
   Compiling shlex v2.0.1
   Compiling pin-project-lite v0.2.17
   Compiling scopeguard v1.2.0
   Compiling bitflags v2.13.2
   Compiling getrandom v0.3.4
   Compiling pkg-config v0.3.34
   Compiling futures-core v0.3.34
   Compiling stable_deref_trait v1.2.1
   Compiling allocator-api2 v0.2.21
   Compiling foldhash v0.1.5
   Compiling hashbrown v0.17.1
   Compiling futures-sink v0.3.34
   Compiling lock_api v0.4.14
   Compiling ahash v0.8.12
   Compiling iana-time-zone v0.1.65
   Compiling slab v0.4.12
   Compiling arrow-schema v53.4.1
   Compiling byteorder v1.5.0
   Compiling typenum v1.20.1
   Compiling regex-syntax v0.8.11
   Compiling syn v3.0.6
   Compiling syn v2.0.119
   Compiling tracing-core v0.1.36
   Compiling vcpkg v0.2.15
   Compiling lexical-util v1.0.7
   Compiling futures-task v0.3.34
   Compiling ryu v1.0.23
   Compiling rustversion v1.0.23
   Compiling subtle v2.6.1
   Compiling crossbeam-utils v0.8.23
   Compiling writeable v0.6.4
   Compiling percent-encoding v2.3.2
   Compiling litemap v0.8.3
   Compiling serde_json v1.0.151
   Compiling zmij v1.0.23
   Compiling icu_normalizer_data v2.3.0
   Compiling utf8_iter v1.0.4
   Compiling num-traits v0.2.19
   Compiling hashbrown v0.15.5
   Compiling base64 v0.22.1
   Compiling tower-service v0.3.3
   Compiling icu_properties_data v2.3.0
   Compiling aho-corasick v1.1.5
   Compiling http v1.5.0
   Compiling httparse v1.10.1
   Compiling utf8parse v0.2.2
   Compiling fnv v1.0.7
   Compiling tower-layer v0.3.3
   Compiling atomic-waker v1.1.2
   Compiling lexical-write-integer v1.0.6
   Compiling lexical-parse-integer v1.0.6
   Compiling crossbeam-epoch v0.9.21
   Compiling indexmap v2.14.2
   Compiling try-lock v0.2.5
   Compiling foreign-types-shared v0.1.1
   Compiling semver v1.0.28
   Compiling linux-raw-sys v0.12.1
   Compiling cpufeatures v0.2.17
   Compiling openssl v0.10.81
   Compiling lazy_static v1.5.1
   Compiling httpdate v1.0.3
   Compiling lexical-parse-float v1.0.6
   Compiling want v0.3.2
   Compiling rustc_version v0.4.1
   Compiling foreign-types v0.3.2
   Compiling anstyle-parse v1.0.0
   Compiling generic-array v0.14.7
   Compiling form_urlencoded v1.2.2
   Compiling sync_wrapper v1.0.2
   Compiling futures-channel v0.3.34
   Compiling lexical-write-float v1.0.6
   Compiling either v1.18.0
   Compiling crossbeam-deque v0.8.8
   Compiling paste v1.0.15
   Compiling is_terminal_polyfill v1.70.2
   Compiling colorchoice v1.0.5
   Compiling anstyle v1.0.14
   Compiling native-tls v0.2.18
   Compiling jobserver v0.1.35
   Compiling mime v0.3.17
   Compiling rustix v1.1.5
   Compiling anstyle-query v1.1.5
   Compiling flatbuffers v24.12.23
   Compiling proc-macro2-diagnostics v0.10.1
   Compiling rayon-core v1.13.0
   Compiling anstream v1.0.0
   Compiling cc v1.6.0
   Compiling openssl-probe v0.2.1
   Compiling ipnet v2.12.2
   Compiling cfg_aliases v0.2.2
   Compiling crc32fast v1.5.2
   Compiling lexical-core v1.0.6
   Compiling base64 v0.23.1
   Compiling thiserror v2.0.21
   Compiling encoding_rs v0.8.42
   Compiling http-body v1.1.0
   Compiling nix v0.30.1
   Compiling bytemuck v1.25.2
   Compiling winnow v0.7.15
   Compiling adler2 v2.0.1
   Compiling utf-8 v0.7.6
   Compiling ucd-trie v0.1.7
   Compiling block-buffer v0.10.4
   Compiling crypto-common v0.1.7
   Compiling http-body-util v0.1.5
   Compiling alloc-no-stdlib v2.0.4
   Compiling toml_write v0.1.2
   Compiling zstd-safe v7.2.1
   Compiling jiff-core v0.1.1
   Compiling simd-adler32 v0.3.10
   Compiling cfg_aliases v0.1.1
   Compiling digest v0.10.7
   Compiling data-encoding v2.11.1
   Compiling same-file v1.0.6
   Compiling unicase v2.10.0
   Compiling timely_container v0.27.0
   Compiling unicode-width v0.2.2
   Compiling yansi v1.0.1
   Compiling bitflags v1.3.2
   Compiling nix v0.28.0
   Compiling walkdir v2.5.0
   Compiling timely_logging v0.27.0
   Compiling miniz_oxide v0.9.1
   Compiling alloc-stdlib v0.2.4
   Compiling errno v0.3.14
   Compiling socket2 v0.6.5
   Compiling signal-hook-registry v1.4.8
   Compiling mio v1.2.4
   Compiling regex-automata v0.4.18
   Compiling getrandom v0.2.17
   Compiling sha1 v0.10.7
   Compiling pest v2.9.2
   Compiling getopts v0.2.24
   Compiling mime_guess v2.0.5
   Compiling anyhow v1.0.104
   Compiling num-integer v0.1.47
   Compiling num-complex v0.4.6
   Compiling rand_core v0.6.4
   Compiling atoi v2.0.0
   Compiling rand_core v0.9.5
   Compiling csv-core v0.1.13
   Compiling uncased v0.9.10
   Compiling num-bigint v0.4.8
   Compiling num-iter v0.1.46
   Compiling jiff v0.2.37
   Compiling core_detect v1.0.0
   Compiling simdutf8 v0.1.5
   Compiling zeroize v1.9.0
   Compiling multiversion_no_op v1.0.0
   Compiling snap v1.1.2
   Compiling timely_bytes v0.27.0
   Compiling csv v1.4.0
   Compiling pest_meta v2.9.2
   Compiling synstructure v0.14.0
   Compiling xattr v1.6.1
   Compiling rayon v1.12.0
   Compiling rustls-pki-types v1.15.1
   Compiling flate2 v1.1.10
   Compiling ordered-float v2.10.1
   Compiling brotli-decompressor v4.0.3
   Compiling cpu-time v1.0.0
   Compiling filetime v0.2.29
   Compiling num_cpus v1.17.0
   Compiling itertools v0.14.0
   Compiling sharded-slab v0.1.7
   Compiling tracing-log v0.2.0
   Compiling combine v4.6.8
   Compiling figment v0.10.19
   Compiling thread_local v1.1.10
   Compiling strsim v0.11.1
   Compiling static_assertions v1.1.0
   Compiling nu-ansi-term v0.50.3
   Compiling twox-hash v2.1.5
   Compiling time-core v0.1.9
   Compiling base64ct v1.8.3
   Compiling num-conv v0.2.2
   Compiling clap_lex v1.1.1
   Compiling integer-encoding v3.0.4
   Compiling num-rational v0.4.2
   Compiling openssl-sys v0.9.117
   Compiling zstd-sys v2.0.13+zstd.1.5.6
   Compiling tikv-jemalloc-sys v0.6.1+5.3.0-1-ge13ca993e8ccb9ba9847cc330696e02839f328f7
   Compiling endian-type v0.1.2
   Compiling deranged v0.5.8
   Compiling inlinable_string v0.1.15
   Compiling num v0.4.3
   Compiling powerfmt v0.2.1
   Compiling password-hash v0.5.0
   Compiling thrift v0.17.0
   Compiling clap_builder v4.6.7
   Compiling brotli v7.0.0
   Compiling lz4_flex v0.11.6
   Compiling regex v1.13.1
   Compiling matchers v0.2.0
   Compiling twox-hash v1.6.3
   Compiling pest_generator v2.9.2
   Compiling tar v0.4.46
   Compiling serde_derive v1.0.229
   Compiling zerofrom-derive v0.1.8
   Compiling yoke-derive v0.8.4
   Compiling tokio-macros v2.7.2
   Compiling zerovec-derive v0.11.6
   Compiling displaydoc v0.2.7
   Compiling zerocopy-derive v0.8.60
   Compiling tracing-attributes v0.1.31
   Compiling futures-macro v0.3.34
   Compiling thiserror-impl v1.0.69
   Compiling openssl-macros v0.1.1
   Compiling columnar_derive v0.11.1
   Compiling thiserror-impl v2.0.21
   Compiling env_filter v2.0.0
   Compiling enum-as-inner v0.6.1
   Compiling pear_codegen v0.2.9
   Compiling async-trait v0.1.92
   Compiling time v0.3.55
   Compiling clap_derive v4.6.7
   Compiling fd-lock v4.0.4
   Compiling getrandom v0.4.3
   Compiling futures-util v0.3.34
   Compiling env_logger v0.11.11
   Compiling serde_path_to_error v0.1.20
   Compiling blake2 v0.10.6
   Compiling sha2 v0.10.9
   Compiling crossbeam-channel v0.5.17
   Compiling symlink v0.1.0
   Compiling home v0.5.12
   Compiling http-range-header v0.4.2
   Compiling hashbrown v0.14.5
   Compiling unicode-width v0.1.14
   Compiling seq-macro v0.3.6
   Compiling zerofrom v0.1.8
   Compiling anndists v0.1.5
   Compiling thiserror v1.0.69
   Compiling matchit v0.7.3
   Compiling tracing v0.1.44
   Compiling sysctl v0.6.0
   Compiling yoke v0.8.3
   Compiling pear v0.2.9
   Compiling unicode-segmentation v1.13.3
   Compiling argon2 v0.5.3
   Compiling uuid v1.26.1
   Compiling pest_derive v2.9.2
   Compiling fs4 v1.1.0
   Compiling zerovec v0.11.8
   Compiling zerotrie v0.2.5
   Compiling mmap-rs v0.7.0
   Compiling arc-swap v1.9.2
   Compiling serde v1.0.229
   Compiling clap v4.6.7
   Compiling tinystr v0.8.4
   Compiling potential_utf v0.1.6
   Compiling icu_locale_core v2.3.0
   Compiling smallvec v1.16.2
   Compiling chrono v0.4.39
   Compiling serde_spanned v0.6.9
   Compiling toml_datetime v0.6.11
   Compiling serde_urlencoded v0.7.1
   Compiling bincode v1.3.3
   Compiling tracing-serde v0.2.0
   Compiling inputlayer-ws-protocol v0.1.0 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/ws-protocol)
   Compiling icu_collections v2.3.0
   Compiling axum-core v0.4.5
   Compiling toml_edit v0.22.27
   Compiling parking_lot_core v0.9.12
   Compiling columnar v0.11.1
   Compiling columnation v0.1.2
   Compiling nibble_vec v0.1.0
   Compiling tracing-subscriber v0.3.23
   Compiling parking_lot v0.12.5
   Compiling dashmap v5.5.3
   Compiling radix_trie v0.2.1
   Compiling tokio v1.53.2
   Compiling icu_provider v2.3.1
   Compiling rustyline v14.0.0
   Compiling timely_communication v0.27.0
   Compiling icu_normalizer v2.3.0
   Compiling icu_properties v2.3.0
   Compiling half v2.7.1
   Compiling ppv-lite86 v0.2.21
   Compiling toml v0.8.23
   Compiling arrow-buffer v53.4.1
   Compiling idna_adapter v1.2.2
   Compiling timely v0.27.0
   Compiling tracing-appender v0.2.5
   Compiling rand_chacha v0.3.1
   Compiling rand_chacha v0.9.0
   Compiling idna v1.1.0
   Compiling rand v0.8.8
   Compiling rand v0.9.5
   Compiling arrow-data v53.4.1
   Compiling url v2.5.8
   Compiling tungstenite v0.24.0
   Compiling arrow-array v53.4.1
   Compiling hnsw_rs v0.3.4 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/vendor/hnsw_rs)
   Compiling tokio-util v0.7.19
   Compiling tower v0.5.3
   Compiling tokio-native-tls v0.3.1
   Compiling tokio-tungstenite v0.24.0
   Compiling tower-http v0.6.11
   Compiling h2 v0.4.19
   Compiling tower-http v0.5.2
   Compiling differential-dataflow v0.20.0
   Compiling arrow-select v53.4.1
   Compiling arrow-row v53.4.1
   Compiling arrow-arith v53.4.1
   Compiling arrow-cast v53.4.1
   Compiling arrow-ord v53.4.1
   Compiling arrow-string v53.4.1
   Compiling hyper v1.11.1
   Compiling arrow-ipc v53.4.1
   Compiling arrow-json v53.4.1
   Compiling arrow-csv v53.4.1
   Compiling zstd v0.13.3
   Compiling hyper-util v0.1.21
   Compiling arrow v53.4.1
   Compiling parquet v53.4.1
   Compiling hyper-tls v0.6.0
   Compiling axum v0.7.9
   Compiling reqwest v0.12.28
   Compiling inputlayer-ontology-client v0.1.0 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/ontology-client)
   Compiling tikv-jemallocator v0.6.1
   Compiling inputlayer v0.1.0 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ)
    Finished `release` profile [optimized] target(s) in 3m 34s
=== Benchmark (1de89a44c7c346d3a55671e55b2100ab5219dcce) ===
baseline    1 sessions: cpu 0.16 cores, write->delta p50/p99 23.8/42.1 ms under load, 0.9 receipts/s, 0 late (max NaN ms), 0 missing, 0 stray
candidate    1 sessions: cpu 0.17 cores, write->delta p50/p99 20.8/29.9 ms under load, 1.0 receipts/s, 0 late (max NaN ms), 0 missing, 0 stray
baseline    5 sessions: cpu 0.94 cores, write->delta p50/p99 27.3/41.5 ms under load, 5.1 receipts/s, 0 late (max NaN ms), 0 missing, 0 stray
candidate    5 sessions: cpu 0.88 cores, write->delta p50/p99 89.3/295.5 ms under load, 5.7 receipts/s, 0 late (max NaN ms), 0 missing, 0 stray
# Session scale (PerSession)

- baseline: 527bacc306ae480504adbdec56c6823672629793 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/baseline-servers/527bacc306ae480504adbdec56c6823672629793-8dc912acd5bf/inputlayer-server)
- candidate: 1de89a44c7c346d3a55671e55b2100ab5219dcce (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/bench-sessions/runs/20261006T064517Z/inputlayer-server)
- load: 1 receipt(s)/s per session for 2 s after a 2 s warm-up; host computeinstance-e00w59r9a240k9qz1e (32 CPUs), load average 40.94 35.06 36.27 44/1629 1899806 -> 45.84 36.70 36.77 129/2167 1927974

| Arm | Sessions | Receipts/s | Server CPU (cores) | No load: write->delta p50 / p99 | Under load: write->delta p50 / p95 / p99 | Receipt commit p50 / p99 | Late (max) / missing / stray |
|---|---|---|---|---|---|---|---|
| baseline | 1 | 0.9 | 0.16 | 42.7 / 108.1 ms | 23.8 / 42.1 / 42.1 ms | 3.0 / 5.2 ms | 0 (NaN ms) / 0 / 0 |
| candidate | 1 | 1.0 | 0.17 | 22.5 / 39.3 ms | 20.8 / 29.9 / 29.9 ms | 3.1 / 3.3 ms | 0 (NaN ms) / 0 / 0 |
| baseline | 5 | 5.1 | 0.94 | 23.3 / 50.4 ms | 27.3 / 41.5 / 41.5 ms | 5.3 / 14.8 ms | 0 (NaN ms) / 0 / 0 |
| candidate | 5 | 5.7 | 0.88 | 74.4 / 151.0 ms | 89.3 / 295.5 / 295.5 ms | 8.0 / 30.1 ms | 0 (NaN ms) / 0 / 0 |
result file: ~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/bench-sessions/runs/20261006T064517Z/result.json

Summary: ~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/bench-sessions/runs/20261006T064517Z/summary.md
++ echo exit=0
exit=0
  • Evidence: Views benchmark with --baseline-rev 527bacc (local file: ~/.no-mistakes/evidence/01M47TEA9EZ3HX2FPK9AJ26GTJ/round3/bench-views.log)
  • Outcome: ⚠️ 0 issues across 3 runs (51m18s)

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info
  • ⚠️ scripts/perf-gate-baseline.sh:36 - The per-baseline work directory (target/perf-gate/baseline-build/&lt;sha&gt;-&lt;toolchain&gt;, an extracted tree plus a full release target dir) is removed only on the success path at line 47. Under set -e, a baseline that fails to compile, or a run that is killed mid-build, leaves it behind; it is reclaimed only if the same commit and toolchain is requested again (line 37). The old shared target/perf-gate/build was bounded at one directory; now a bisect over N commits that do not build leaves N full target directories on the benchmark host. Remedy: set trap &#39;rm -rf &#34;$WORK&#34;&#39; EXIT once WORK is created, and drop the explicit rm -rf at line 47. The selftest's "no build directory is left behind" check (scripts/perf-gate-baseline-selftest.sh:103) covers only the success path.
  • ⚠️ scripts/perf-gate-baseline.sh:44 - Simplification: the copy-to-$BIN.tmp-then-mv at lines 44-46 is a new component that guards against an interrupted copy being taken for a cached binary. No intent requirement needs it: the intent is that each baseline is built from its own sources, which the per-baseline source and target directory already delivers, and the previous code used a plain cp. Recommend reverting to a single cp into $BIN, unless the author wants to keep this extra hardening.
  • ℹ️ scripts/perf-gate-baseline.sh:32 - Moving the cache to baseline-servers/ correctly stops any binary built in the shared directory from being read again, but nothing removes the existing target/perf-gate/build, target/perf-gate/src and target/perf-gate/servers/&lt;sha&gt;-* on the benchmark host's clone. They are now dead disk (servers/candidate is still used by scripts/perf-gate.sh:74) and need a one-off manual cleanup. Commits that predate this change still run their own old script when checked out on the host, so they keep using the shared directory; they cannot poison the new cache.

🔧 Fix applied.
1 info still open:

  • ℹ️ scripts/perf-gate-baseline.sh:32 - Moving the cache to baseline-servers/ correctly stops any binary built in the shared directory from being read again, but nothing removes the existing target/perf-gate/build, target/perf-gate/src and target/perf-gate/servers/&lt;sha&gt;-* on the benchmark host's clone. They are now dead disk (servers/candidate is still used by scripts/perf-gate.sh:74) and need a one-off manual cleanup. Commits that predate this change still run their own old script when checked out on the host, so they keep using the shared directory; they cannot poison the new cache.

  • ℹ️ scripts/perf-gate-baseline.sh:43 - Tradeoff, no action needed: because each baseline gets its own target directory, removed by the EXIT trap, every uncached baseline is now a cold release build with all features, third-party dependencies included. The old shared target/perf-gate/build kept those compiled between baselines, so a bisect over N uncached commits now pays N full builds on the benchmark host. The intent names a per-baseline CARGO_TARGET_DIR as an accepted fix, and finished binaries are still cached per commit and toolchain in baseline-servers/, so only the first build of each commit pays.

⚠️ **Test** - 0 issues

✅ No issues found.

  • Live validation: ✅ go - 8 of 8 scenarios driven live against the product
Scenario Result Live Evidence
Build the real #387 baseline, then its older parent, on one clone: each binary is its own commit's server ✅ pass live real-newer-then-older.txt and real-binaries-compared.txt: both builds exit 0; 527bacc binary sha256 4382b65d… with 0 access_denied strings, efa32ce binary cc89631e… with 10
Same order with the base commit's shared build directory reproduces the bug: the older baseline comes back as the newer server ✅ pass live real-old-newer-then-older.txt and real-binaries-compared.txt: the 527bacc 'build' finishes in 3 s with exit 0 and its binary is byte-identical to #387's (cc89631e…, 10 access_denied strings)
Fixture workspace, both orders (newer then older, older then newer): each baseline prints its own engine and protocol version ✅ pass live selftest-fixed.txt (all PASS); selftest-old-shared-dir.txt shows the same checks failing on the pre-change build ('engine-newer protocol-newer' for the older baseline, and the reverse)
Asking for an already built baseline returns the cached binary without rebuilding ✅ pass live selftest-fixed.txt ('a cached baseline is not built again'); gate-aa-quick.txt has no 'Build baseline server' line for the cached 527bacc
A baseline that does not compile fails with non-zero status, caches nothing and leaves no build directory ✅ pass live selftest-fixed.txt: 'a baseline that does not compile fails', 'no build directory is left behind, built or not'
Interrupting a baseline build mid-flight (SIGINT or SIGTERM) removes its build directory and caches no binary ✅ pass live interrupt-cleanup.txt: baseline-build holds the commit's directory during the build and is empty after each signal; baseline-servers is empty
Run the perf gate end to end against a baseline commit: it measures the helper's binary for that commit ✅ pass live gate-aa-quick.txt: exit 0, summary reports baseline 527bacc with binary sha256 4382b65d6c57, the same binary the helper built for that commit
Run the views benchmark with --baseline-rev: the baseline is built by the helper and measured next to the candidate ✅ pass live bench-views-baseline.txt: exit 0, 'Build baseline server 527bacc…', baseline path under target/perf-gate/baseline-servers/, both arms measured

  • scripts/perf-gate-baseline-selftest.sh (fixed helper: all 8 checks pass)

  • scripts/perf-gate-baseline-selftest.sh in a scratch repo whose scripts/perf-gate-baseline.sh is the base commit's shared-target/perf-gate/build block (3 checks fail: reproduces the bug)

  • scripts/perf-gate-baseline.sh efa32cec then scripts/perf-gate-baseline.sh 527bacc3 on the real repo (170 s and 189 s full release builds, both exit 0, baseline-build/ empty after each)

  • old-shared-dir-baseline-shim.sh efa32cec then 527bacc3 on the real repo (second 'build' finishes in 3 s and returns the #387 binary)

  • sha256sum and grep -c -a access_denied over the four resulting inputlayer-server binaries

  • interrupt-check.sh scripts/perf-gate-baseline.sh: timeout -s INT 8 and timeout -s TERM 8 on a baseline build held open by a sleeping build script

  • scripts/perf-gate.sh --aa --baseline-rev 527bacc3 --profile quick --rounds 1 --fixtures cheap_query --no-verdict (exit 0)

  • scripts/bench-views.sh --baseline-rev 527bacc3 --edges 1000 --writes 3 --large-writes 2 --reads 2 --keyed 1 --recursive-keyed 1 --large-recursive-keyed 1 --filler-rules 5 (exit 0)

  • ⚠️ scripts/perf-gate-baseline.sh:38 - A baseline build killed with SIGTERM or SIGHUP often leaves target/perf-gate/baseline-build/&lt;sha&gt;-&lt;toolchain&gt; behind (5 of 8 tries on a loaded host; SIGINT was clean 4 of 4), against the recorded decision to remove it on interruption too. Bash runs the EXIT trap as soon as the signal arrives, so rm -rf races cargo and rustc, which are still writing. Impact is disk only: no wrong binary is cached, and the directory is wiped if the same commit is requested again. Adding trap &#39;exit 143&#39; TERM HUP after the EXIT trap makes bash wait for cargo to exit first; a copy of the helper with that one line was clean in 9 of 9 tries (3 per signal) under the same load (interrupt-repeat-with-signal-trap.log). The selftest covers only the compile-failure path, not interruption.

  • 🚨 live validation verdict: no-go (7 of 7 scenarios were driven live against the product); failed: Kill a baseline build with SIGTERM or SIGHUP (to the process group): no build directory is left behind

  • Live validation: ❌ no-go - 7 of 7 scenarios driven live against the product

Scenario Result Live Evidence
On the real repository, build the baseline of #387 (efa32ce) and then of its older parent (527bacc): both build and each server is its own commit's ✅ pass live real-repo-summary.txt, real-new-newer.log, real-new-older.log: both exit 0 with a full 3m43s build each; the binaries differ (cc89631e vs 4382b65d) and only the #387 one contains the access_denied c…
Regression reproduced: the base commit's shared build directory, same two commits in the same order, hands back the wrong server for the older one ✅ pass live real-repo-summary.txt, real-old-older.log: the older commit 'finished' in 0.77s and its cached binary is byte-identical to the #387 server (cc89631e, access_denied present)
Fixture workspace, both build orders (newer then older, older then newer): each baseline prints its own engine and protocol; a cached baseline is not rebuilt; a commit that does not compile fails and… ✅ pass live selftest-target.log (scripts/perf-gate-baseline-selftest.sh, all 8 checks pass); selftest-base-logic.log (same checks against the base commit's build block: 3 fail, e.g. the older baseline says 'eng…
Interrupt a baseline build with Ctrl-C (SIGINT to the process group): no build directory is left behind ✅ pass live interrupt-repeat.log: SIGINT 4 of 4 clean; interrupt.log
Kill a baseline build with SIGTERM or SIGHUP (to the process group): no build directory is left behind ❌ fail live interrupt-repeat.log: SIGTERM left the directory in 3 of 4 tries, SIGHUP in 2 of 4 (a partial target/release tree remains, src is gone); interrupt.log shows one more SIGTERM leftover
Run the perf gate end to end against the older baseline (scripts/perf-gate.sh --baseline-rev efa32cec~1, quick profile): it measures both arms using the helper's cached baseline and writes a report ✅ pass live gate-quick.log: every fixture measured on both arms with baseline 527bacc, report written; exit status 3 (INVALID) is the quick profile's expected outcome with 2 rounds, not a failure of the change
Run the views and sessions benchmarks with --baseline-rev efa32cec~1: both compare the working tree against the helper's baseline binary ✅ pass live bench-views.log and bench-sessions.log: both exit 0 and name baseline 527bacc at target/perf-gate/baseline-servers/527bacc3...-8dc912acd5bf/inputlayer-server
  • scripts/perf-gate-baseline-selftest.sh at the target commit (all checks pass)
  • The same selftest with the helper swapped for the base commit's build block from scripts/perf-gate.sh at d5df1ba3 (3 checks fail, reproducing the bug)
  • scripts/perf-gate-baseline.sh efa32cec then scripts/perf-gate-baseline.sh efa32cec~1 on the real repository, comparing the two binaries by hash and by presence of the access_denied code
  • The base commit's build block on the same two commits in the same order (older baseline comes back byte-identical to the newer one)
  • timeout -s INT|TERM|HUP 8 scripts/perf-gate-baseline.sh HEAD on a slow-building fixture repository, 4 times per signal, listing target/perf-gate/baseline-build afterwards
  • The same interruption loop against a copy of the helper with trap &#39;exit 143&#39; TERM HUP added (3 per signal, all clean)
  • scripts/perf-gate.sh --baseline-rev efa32cec~1 --candidate-bin <efa32cec baseline> --profile quick --rounds 2
  • scripts/bench-views.sh --baseline-rev efa32cec~1 --edges 1000 --writes 3 --reads 2 --keyed 1 --recursive-keyed 1 --filler-rules 5
  • scripts/bench-sessions.sh --baseline-rev efa32cec~1 --sessions 1,5 --load-secs 2

🔧 Fix applied.
✅ Re-checked - no issues remain.

  • Live validation: ✅ go - 8 of 8 scenarios driven live against the product
Scenario Result Live Evidence
Build the baseline of #387 (efa32ce) and then of its older parent (527bacc) on the real repository: each server is its own commit's ✅ pass live round3/real-new.txt: both exit 0, each build compiles inputlayer-ws-protocol itself; efa32ce sha256 cc89631e with 10 access_denied strings, 527bacc sha256 4382b65d with 0; baseline-build empty aft…
The same two builds with the base commit's shared build directory reproduce the reported fault: the older baseline is the newer commit's server ✅ pass live round3/real-old.txt: the 527bacc build finishes in 0.25s without compiling ws-protocol, and its binary is byte-identical to efa32ce's (sha256 cc89631e, 10 access_denied strings)
Fixture workspace, newer-then-older and older-then-newer: each baseline server prints its own commit's engine and protocol; the base logic does not ✅ pass live round3/selftest-target.log (all PASS, exit 0) against round3/selftest-base-logic.log (2 FAIL: 'engine-newer protocol-newer' for the older baseline and the reverse, exit 1)
Request an already built baseline again: it is returned from the cache without a rebuild ✅ pass live round3/selftest-target.log 'a cached baseline is not built again'; round3/gate-quick.log, bench-sessions.log and bench-views.log contain no 'Build baseline server' for the cached 527bacc
Request a baseline that does not compile: the helper fails and leaves no build directory behind ✅ pass live round3/selftest-target.log: 'a baseline that does not compile fails' and 'no build directory is left behind, built or not'
Kill a baseline build with SIGINT, SIGTERM or SIGHUP, to the process group or to the helper alone: no build directory is left behind and no partial binary is cached ✅ pass live round3/interrupt-group.log: 9 of 9 clean under load average about 30; round3/interrupt-group-without-signal-trap.log: the helper without the trap leaves the directory in 4 of 6 SIGTERM/SIGHUP tries; r…
Run the perf gate against a baseline commit: it gets the baseline from the helper and measures both servers ✅ pass live round3/gate-quick.log: baseline 527bacc against the efa32ce server, 8 fixture runs over 2 rounds, report written; exit 3 with 'Performance gate: INVALID' is what the quick profile always gives
Run the session and views benchmarks with --baseline-rev: both measure the helper's baseline binary beside the working tree's ✅ pass live round3/bench-sessions.log and round3/bench-views.log: baseline path is target/perf-gate/baseline-servers/527bacc3...-8dc912acd5bf/inputlayer-server, both exit 0
  • scripts/perf-gate-baseline-selftest.sh at 1de89a44: all 10 checks pass
  • tools/selftest-against-old.sh (same selftest against the base commit's shared target/perf-gate/build logic): fails the two second-built baselines
  • round3/selftest-without-signal-trap.sh (the selftest against the helper minus the trap &#39;exit 143&#39; TERM HUP line): fails the SIGTERM and SIGHUP checks
  • scripts/perf-gate-baseline.sh efa32cec then scripts/perf-gate-baseline.sh 527bacc3 on the real repository, comparing sha256 and access_denied string counts of the two servers
  • tools/old-baseline.sh efa32cec then tools/old-baseline.sh 527bacc3 (base-commit logic) on the real repository
  • tools/interrupt-repeat.sh scripts/perf-gate-baseline.sh 3: SIGINT, SIGTERM and SIGHUP to the helper's process group 8s into a build, 3 tries each; the same against the helper without the signal trap
  • round3/interrupt-helper-only.sh scripts/perf-gate-baseline.sh: SIGTERM, SIGHUP and SIGINT to the helper process alone mid-build
  • scripts/perf-gate.sh --baseline-rev 527bacc3 --candidate-bin <efa32cec baseline> --profile quick --rounds 2 --fixtures cheap_query,insert_single
  • scripts/bench-sessions.sh --baseline-rev 527bacc3 --sessions 1,5 --load-secs 2
  • scripts/bench-views.sh --baseline-rev 527bacc3 --edges 1000 --writes 3 --large-writes 3 --reads 2 --keyed 1 --recursive-keyed 1 --large-recursive-keyed 1 --filler-rules 5
  • python3 -m unittest scripts/test_pre_pr.py
✅ **Document** - passed

✅ No issues found.

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

✅ No issues found.

jsam added 4 commits October 6, 2026 06:13
@jsam
fix(perf-gate): build every baseline in a target directory of its own
f0eea52 
scripts/perf-gate.sh, bench-sessions.sh and bench-views.sh extracted each
baseline with `git archive` and built all of them in one shared
target/perf-gate/build. Cargo names a workspace crate's artifacts by its path
inside the workspace, not by where the workspace is, and rebuilds one only
when a source file is newer than the last build. `git archive` dates every
file at its commit, which is older than any build already in the directory,
so a later baseline reused an earlier one's workspace crates: the build
failed where the APIs had diverged (`missing field code in ServerFrame`
after #387), linked another commit's `ws-protocol` or `ontology-client`
where they had not, and returned the earlier baseline's whole server where
no file had been added or removed between the two commits.

The three scripts now build baselines through scripts/perf-gate-baseline.sh:
each baseline gets a source and target directory of its own under
target/perf-gate/baseline-build/, removed once the binary is cached. The
cache moves to target/perf-gate/baseline-servers/, so a binary cached by the
shared build is never read again, and the three scripts now share one
toolchain key (the gate hashed `rustc -V` without its newline, the
benchmarks with it).

scripts/perf-gate-baseline-selftest.sh builds two baselines that differ in
the root crate and in a path dependency, in both orders, and checks each
binary reports its own commit. It runs in `make pre-pr-selftest` and in the
full suite's perf-gate tool job.
@jsam
no-mistakes(review): clean baseline build dir on failure; plain cp
82b0918
@jsam
no-mistakes(document): docs(perf-gate): baseline build dir is removed…
2280c37 
… on failure too
@jsam
no-mistakes(test): fix(perf-gate): remove baseline build dir on SIGTE…
1de89a4 
…RM/SIGHUP
@jsam

jsam commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Moved to https://github.com/inputlayer/inputlayer-symbolic/pull/8

@jsam jsam closed this Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

See More

Successfully merging this pull request may close these issues.

1 participant

@jsam