fix(perf-gate): build every baseline in a target directory of its own by jsam · Pull Request #399 · inputlayer/inputlayer

jsam · 2026-10-06T05:59:09Z

Intent

Performance regressions and the perf gate in inputlayer/inputlayer are part of getting the engine production ready, and benchmarks run only on the benchmark host. The perf gate there currently produces wrong or failed baseline builds, found while bisecting #374:

scripts/perf-gate.sh extracts each baseline with git archive | tar -x (file mtimes = the commit's time) and builds every baseline in one shared target/perf-gate/build. Cargo's metadata hash does not include the path of a path dependency, so every baseline's inputlayer-ws-protocol maps to the same artifact, and an older baseline extracted after a newer one was built sees files older than the fingerprint and reuses the newer crate. The build fails when the APIs diverged (missing field code in ServerFrame, from #387 built by another lane's run on the shared host clone), and silently links the wrong crate when they did not. Five bisect steps failed this way. Fix: touch the extracted tree, or a per-baseline CARGO_TARGET_DIR. Until then only baselines at or after the newest-built ws-protocol build.

Fix it so every baseline is built from exactly its own sources. Deliver it as a small PR; done means merged with CI green.

What Changed

New scripts/perf-gate-baseline.sh builds a baseline server from git archive of its commit in its own CARGO_TARGET_DIR (target/perf-gate/baseline-build/<commit>-<toolchain>/), removed when the build ends whether it succeeded or not, so cargo can no longer reuse another commit's workspace crates. scripts/perf-gate.sh, scripts/bench-sessions.sh and scripts/bench-views.sh now call it instead of each carrying a copy of the build that shared target/perf-gate/build.
The baseline cache moves from target/perf-gate/servers/ to target/perf-gate/baseline-servers/, and the cache path and key in .github/workflows/full-suite.yml change with it, so binaries built in the shared directory are not picked up again.
New scripts/perf-gate-baseline-selftest.sh builds two commits of a throwaway workspace in both orders and checks that each binary reports its own commit's sources, that a cached baseline is not rebuilt, and that a commit that does not compile fails and leaves no build directory behind. It runs from make pre-pr-selftest and the perf-gate-tool CI job; scripts/test_pre_pr.py, perf-gate/README.md, TESTING.md, CONTRIBUTING and the Makefile comment are updated to match.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change is confined to benchmark tooling and durably removes the shared target directory from all three callers (gate, sessions, views) through one helper with a behavioural regression test; the only defect found is a disk leak when a baseline build fails.

Testing

No baseline commands had run before this step. I ran the new selftest against the fixed helper (all checks pass) and against the base commit's shared-directory build (3 checks fail, each naming the other commit's sources). On the real repository I built #387 (efa32ce) and then its parent (527bacc) with the fixed helper: two distinct binaries, only the #387 one containing access_denied, and no build directory left behind. The same order with the base commit's build returned a byte-identical #387 binary for the parent in 3 seconds with exit 0 — the silent wrong-binary mode, not the compile error from the report. I interrupted a build mid-flight with SIGINT and SIGTERM (build directory removed, nothing cached), ran scripts/perf-gate.sh as a quick A/A smoke on the parent baseline, and ran a tiny scripts/bench-views.sh --baseline-rev. All scenarios pass; build output was removed from the worktree afterwards. No UI surface, so no visual evidence.

Live validation: ✅ go - 8 of 8 scenarios driven live against the product

Scenario	Result	Live	Evidence
Build the real #387 baseline, then its older parent, on one clone: each binary is its own commit's server	✅ pass	live	real-newer-then-older.txt and real-binaries-compared.txt: both builds exit 0; `527bacc` binary sha256 4382b65d… with 0 `access_denied` strings, `efa32ce` binary cc89631e… with 10
Same order with the base commit's shared build directory reproduces the bug: the older baseline comes back as the newer server	✅ pass	live	real-old-newer-then-older.txt and real-binaries-compared.txt: the `527bacc` 'build' finishes in 3 s with exit 0 and its binary is byte-identical to #387's (cc89631e…, 10 `access_denied` strings)
Fixture workspace, both orders (newer then older, older then newer): each baseline prints its own engine and protocol version	✅ pass	live	selftest-fixed.txt (all PASS); selftest-old-shared-dir.txt shows the same checks failing on the pre-change build ('engine-newer protocol-newer' for the older baseline, and the reverse)
Asking for an already built baseline returns the cached binary without rebuilding	✅ pass	live	selftest-fixed.txt ('a cached baseline is not built again'); gate-aa-quick.txt has no 'Build baseline server' line for the cached `527bacc`
A baseline that does not compile fails with non-zero status, caches nothing and leaves no build directory	✅ pass	live	selftest-fixed.txt: 'a baseline that does not compile fails', 'no build directory is left behind, built or not'
Interrupting a baseline build mid-flight (SIGINT or SIGTERM) removes its build directory and caches no binary	✅ pass	live	interrupt-cleanup.txt: `baseline-build` holds the commit's directory during the build and is empty after each signal; `baseline-servers` is empty
Run the perf gate end to end against a baseline commit: it measures the helper's binary for that commit	✅ pass	live	gate-aa-quick.txt: exit 0, summary reports baseline `527bacc` with binary sha256 4382b65d6c57, the same binary the helper built for that commit
Run the views benchmark with --baseline-rev: the baseline is built by the helper and measured next to the candidate	✅ pass	live	bench-views-baseline.txt: exit 0, 'Build baseline server `527bacc`…', baseline path under `target/perf-gate/baseline-servers/`, both arms measured

Evidence: Real repo, this change: #387 then its parent, both built from their own sources

# pre-change shared build directory (target/perf-gate/servers), newer #387 built first, then its parent 527bacc3: servers/527bacc3…-415a6fa19307/ sha256=cc89631edeaa6a8b access_denied strings=10 servers/efa32cec…-415a6fa19307/ sha256=cc89631edeaa6a8b access_denied strings=10 # this change (target/perf-gate/baseline-servers), same order: baseline-servers/527bacc3…-8dc912acd5bf/ sha256=4382b65d6c577d86 access_denied strings=0 baseline-servers/efa32cec…-8dc912acd5bf/ sha256=cc89631edeaa6a8b access_denied strings=10 # 'access_denied' appears in src/ and ws-protocol/ of efa32cec (#387) in 4 files, of its parent 527bacc3 in 0 files

# pre-change shared build directory (target/perf-gate/servers), newer #387 built first, then its parent 527bacc3:
servers/527bacc306ae480504adbdec56c6823672629793-415a6fa19307/ sha256=cc89631edeaa6a8b access_denied strings=10
servers/efa32cec62979f089624c76f6846c9b9e27b4381-415a6fa19307/ sha256=cc89631edeaa6a8b access_denied strings=10
# this change (target/perf-gate/baseline-servers), same order:
baseline-servers/527bacc306ae480504adbdec56c6823672629793-8dc912acd5bf/ sha256=4382b65d6c577d86 access_denied strings=0
baseline-servers/efa32cec62979f089624c76f6846c9b9e27b4381-8dc912acd5bf/ sha256=cc89631edeaa6a8b access_denied strings=10
# 'access_denied' appears in src/ and ws-protocol/ of efa32cec (#387) in 4 files, of its parent 527bacc3 in 0 files

Evidence: Real repo, fixed helper transcript (newer then older)

$ scripts/perf-gate-baseline.sh efa32cec
~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/baseline-servers/efa32cec62979f089624c76f6846c9b9e27b4381-8dc912acd5bf/inputlayer-server
exit=0 seconds=170
leftover baseline-build: []
$ scripts/perf-gate-baseline.sh 527bacc3
~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/baseline-servers/527bacc306ae480504adbdec56c6823672629793-8dc912acd5bf/inputlayer-server
exit=0 seconds=189
leftover baseline-build: []

Evidence: Real repo, pre-change build transcript (older baseline 'built' in 3 s)

$ old-shared-dir-baseline-shim.sh efa32cec   (the base commit's build: one shared target/perf-gate/build)
~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/servers/efa32cec62979f089624c76f6846c9b9e27b4381-415a6fa19307/inputlayer-server
exit=0 seconds=211
$ old-shared-dir-baseline-shim.sh 527bacc3   (the base commit's build: one shared target/perf-gate/build)
~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/servers/527bacc306ae480504adbdec56c6823672629793-415a6fa19307/inputlayer-server
exit=0 seconds=3

Evidence: Selftest against the pre-change shared-directory build (fails)

PASS: the newer baseline is built from its own sources FAIL: the older baseline was built from other sources: it says 'engine-newer protocol-newer' PASS: the older baseline is built from its own sources FAIL: the newer baseline was built from other sources: it says 'engine-older protocol-older' PASS: a cached baseline is not built again PASS: the older baseline is built from its own sources FAIL: a baseline that does not compile was reported built perf-gate-baseline-selftest: 3 failure(s)

PASS: the newer baseline is built from its own sources
FAIL: the older baseline was built from other sources: it says 'engine-newer protocol-newer'
PASS: the older baseline is built from its own sources
FAIL: the newer baseline was built from other sources: it says 'engine-older protocol-older'
PASS: a cached baseline is not built again
PASS: the older baseline is built from its own sources
FAIL: a baseline that does not compile was reported built
ls: cannot access '/tmp/tmp.g8YThuMDNG/older-first/target/perf-gate/baseline-build': No such file or directory
PASS: no build directory is left behind, built or not
perf-gate-baseline-selftest: 3 failure(s)

Evidence: Selftest against this change (passes)

PASS: the newer baseline is built from its own sources
PASS: the older baseline is built from its own sources
PASS: the older baseline is built from its own sources
PASS: the newer baseline is built from its own sources
PASS: a cached baseline is not built again
PASS: the older baseline is built from its own sources
PASS: a baseline that does not compile fails
PASS: no build directory is left behind, built or not
perf-gate-baseline-selftest: all checks passed

Evidence: Pre-change baseline build used for the reproductions

#!/usr/bin/env bash
# Baseline build exactly as scripts/perf-gate.sh did it at base 2ed4a4a3
# (shared target/perf-gate/build), wrapped to print the binary path.
set -euo pipefail
ROOT=$(git rev-parse --show-toplevel)
cd "$ROOT"
OUT=$ROOT/target/perf-gate
BASELINE_SHA=$(git rev-parse --verify "$1^{commit}")
RUSTC=$(rustc -V)
TOOLCHAIN_KEY=$(printf '%s' "$RUSTC" | sha256sum | cut -c1-12)
BASELINE_BIN=$OUT/servers/$BASELINE_SHA-$TOOLCHAIN_KEY/inputlayer-server
if [ ! -x "$BASELINE_BIN" ]; then
    echo "=== Build baseline server $BASELINE_SHA ===" >&2
    SRC=$OUT/src/$BASELINE_SHA
    rm -rf "$SRC" && mkdir -p "$SRC"
    git archive --format=tar "$BASELINE_SHA" | tar -x -C "$SRC"
    if [ -f Cargo.lock ]; then cp Cargo.lock "$SRC/"; fi
    CARGO_TARGET_DIR=$OUT/build cargo build --release --all-features \
        --manifest-path "$SRC/Cargo.toml" --bin inputlayer-server >&2
    mkdir -p "$(dirname "$BASELINE_BIN")"
    cp "$OUT/build/release/inputlayer-server" "$BASELINE_BIN"
fi
echo "$BASELINE_BIN"

Evidence: Interrupted build leaves nothing behind

== SIGINT to the helper's process group, 8s into the build == during the build: baseline-build holds [22740b17…-8dc912acd5bf] helper exit status 124 after 8s after SIGINT: baseline-build holds []; baseline-servers holds [] == SIGTERM to the helper's process group, 8s into the build == during the build: baseline-build holds [22740b17…-8dc912acd5bf] helper exit status 124 after 8s after SIGTERM: baseline-build holds []; baseline-servers holds []

== SIGINT to the helper's process group, 8s into the build ==
during the build: baseline-build holds [22740b17befd5a2865a43a7122180fce3065729c-8dc912acd5bf]
helper exit status 124 after 8s
after SIGINT: baseline-build holds []; baseline-servers holds []
== SIGTERM to the helper's process group, 8s into the build ==
during the build: baseline-build holds [22740b17befd5a2865a43a7122180fce3065729c-8dc912acd5bf]
helper exit status 124 after 8s
after SIGTERM: baseline-build holds []; baseline-servers holds []

Evidence: Interruption driver script

#!/usr/bin/env bash
# Interrupt scripts/perf-gate-baseline.sh while cargo is building and report
# what is left under target/perf-gate/baseline-build. Usage: interrupt-check.sh HELPER
set -uo pipefail
HELPER=$1
T=$(mktemp -d); R=$T/repo; mkdir -p "$R/src/bin"
export GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_NOSYSTEM=1
git -C "$R" init -q; git -C "$R" config user.name t; git -C "$R" config user.email t@invalid
cat > "$R/Cargo.toml" <<'EOF'
[package]
name = "inputlayer"
version = "0.1.0"
edition = "2021"
build = "build.rs"
[[bin]]
name = "inputlayer-server"
path = "src/bin/server.rs"
EOF
# A build script that sleeps keeps the build in flight long enough to interrupt.
echo 'fn main(){ std::thread::sleep(std::time::Duration::from_secs(40)); }' > "$R/build.rs"
echo 'fn main(){ println!("slow"); }' > "$R/src/bin/server.rs"
git -C "$R" add .; git -C "$R" commit -qm slow
cd "$R"
B=$R/target/perf-gate/baseline-build
for sig in INT TERM; do
  echo "== SIG$sig to the helper's process group, 8s into the build =="
  rm -rf "$R/target"
  ( sleep 6; echo "during the build: baseline-build holds [$(ls "$B")]" ) &
  s=$(date +%s)
  timeout -s "$sig" 8 "$HELPER" HEAD >/dev/null 2>"$T/log"
  echo "helper exit status $? after $(( $(date +%s)-s ))s"
  wait
  echo "after SIG$sig: baseline-build holds [$(ls "$B")]; baseline-servers holds [$(ls "$R/target/perf-gate/baseline-servers" 2>/dev/null)]"
done
cd /; rm -rf "$T"

Evidence: Perf gate quick A/A smoke on the helper-built baseline

=== Build perf-gate ===
=== Measure (baseline 527bacc306ae480504adbdec56c6823672629793 vs candidate 527bacc306ae480504adbdec56c6823672629793) ===
round 1/1 cheap_query    baseline  concurrent_latency_us p50 1.40ms p99 11.11ms | latency_us p50 0.76ms p99 5.40ms | queries_per_sec 3999
round 1/1 cheap_query    candidate concurrent_latency_us p50 1.48ms p99 5.50ms | latency_us p50 0.71ms p99 3.51ms | queries_per_sec 4282
run file: ~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/runs/20261006T054731Z/run.json
- server: baseline `527bacc306ae480504adbdec56c6823672629793` (binary sha256 4382b65d6c57); candidate `527bacc306ae480504adbdec56c6823672629793` (binary sha256 4382b65d6c57)
- host `computeinstance-e00w59r9a240k9qz1e`: AMD EPYC-Genoa Processor (32 CPUs, 125 GB), kernel 6.11.0-1016-nvidia, governor unknown, data on ext4, server CPUs unpinned
- profile `quick`, 1 rounds per arm, load average start `46.44 36.33 28.28 15/1590 3501874`, end `46.44 36.33 28.28 26/1410 3502435`
- build: rustc 1.99.0 (b940084d7 2026-09-28); cargo build --release --all-features

| fixture | latency series | p50 | p99 | p50 range over runs | runs | samples |
|---|---|---|---|---|---|---|
| cheap_query | concurrent_latency_us | 1.443 ms | 8.308 ms | 1.403 ms..1.483 ms | 2 | 480 |
| cheap_query | latency_us | 0.735 ms | 4.455 ms | 0.714 ms..0.755 ms | 2 | 160 |

| fixture | rate | per second | range over runs | runs |
|---|---|---|---|---|
| cheap_query | queries_per_sec | 4141 | 3999..4282 | 2 |

| fixture | gauge | value | range over runs | runs |
|---|---|---|---|---|
| cheap_query | server_peak_rss_kb | 50452 | 50080..50824 | 2 |


Summary: ~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/runs/20261006T054731Z/summary.md

real	0m2.506s
user	0m0.989s
sys	0m0.743s

Evidence: Views benchmark with --baseline-rev

=== Build perf-gate and server (release, all features) ===
    Updating crates.io index
     Locking 445 packages to highest Rust 1.88 compatible versions
      Adding argon2 v0.5.3 (available: v0.6.0)
      Adding arrow v53.4.1 (available: v60.0.0)
      Adding axum v0.7.9 (available: v0.8.9)
      Adding base64 v0.22.1 (available: v0.23.1)
      Adding bincode v1.3.3 (available: v3.0.0)
      Adding chrono v0.4.39 (available: v0.4.45)
      Adding criterion v0.5.1 (available: v0.8.2)
      Adding dashmap v5.5.3 (available: v6.2.1)
      Adding differential-dataflow v0.20.0 (available: v0.25.1)
      Adding generic-array v0.14.7 (available: v0.14.9)
      Adding parquet v53.4.1 (available: v60.0.0)
      Adding rand v0.8.8 (available: v0.10.3)
      Adding reqwest v0.12.28 (available: v0.13.5)
      Adding rustyline v14.0.0 (available: v18.0.1)
      Adding sha2 v0.10.9 (available: v0.11.0)
      Adding thiserror v1.0.69 (available: v2.0.21)
      Adding tikv-jemalloc-sys v0.6.1+5.3.0-1-ge13ca993e8ccb9ba9847cc330696e02839f328f7 (available: v0.7.1+5.3.1-0-g81034ce1f1373e37dc865038e1bc8eeecf559ce8)
      Adding tikv-jemallocator v0.6.1 (available: v0.7.0)
      Adding timely v0.27.0 (available: v0.31.0)
      Adding tokio-tungstenite v0.24.0 (available: v0.30.0)
      Adding toml v0.8.23 (available: v1.1.6+spec-1.1.0)
      Adding tower-http v0.5.2 (available: v0.7.1)
      Adding tower-http v0.6.11 (available: v0.7.1)
      Adding uuid v1.26.1 (available: v1.27.0, requires Rust 1.89.0)
      Adding zstd-safe v7.2.1 (available: v7.3.0)
      Adding zstd-sys v2.0.13+zstd.1.5.6 (available: v2.1.0+zstd.1.5.7)
   Compiling cfg-if v1.0.5
   Compiling libc v0.2.190
   Compiling serde_core v1.0.229
   Compiling bytes v1.12.1
   Compiling once_cell v1.21.4
   Compiling autocfg v1.5.1
   Compiling libm v0.2.16
   Compiling zerocopy v0.8.60
   Compiling itoa v1.0.18
   Compiling equivalent v1.0.2
   Compiling memchr v2.8.3
   Compiling log v0.4.34
   Compiling shlex v2.0.1
   Compiling find-msvc-tools v0.1.14
   Compiling pin-project-lite v0.2.17
   Compiling getrandom v0.3.4
   Compiling scopeguard v1.2.0
   Compiling bitflags v2.13.2
   Compiling parking_lot_core v0.9.12
   Compiling pkg-config v0.3.34
   Compiling futures-core v0.3.34
   Compiling stable_deref_trait v1.2.1
   Compiling allocator-api2 v0.2.21
   Compiling hashbrown v0.17.1
   Compiling foldhash v0.1.5
   Compiling futures-sink v0.3.34
   Compiling ahash v0.8.12
   Compiling arrow-schema v53.4.1
   Compiling tracing-core v0.1.36
   Compiling slab v0.4.12
   Compiling iana-time-zone v0.1.65
   Compiling syn v3.0.6
   Compiling syn v2.0.119
   Compiling lock_api v0.4.14
   Compiling byteorder v1.5.0
   Compiling regex-syntax v0.8.11
   Compiling typenum v1.20.1
   Compiling vcpkg v0.2.15
   Compiling ryu v1.0.23
   Compiling futures-task v0.3.34
   Compiling lexical-util v1.0.7
   Compiling num-traits v0.2.19
   Compiling litemap v0.8.3
   Compiling crossbeam-utils v0.8.23
   Compiling percent-encoding v2.3.2
   Compiling hashbrown v0.15.5
   Compiling subtle v2.6.1
   Compiling http v1.5.0
   Compiling serde_json v1.0.151
   Compiling aho-corasick v1.1.5
   Compiling rustversion v1.0.23
   Compiling writeable v0.6.4
   Compiling zmij v1.0.23
   Compiling icu_normalizer_data v2.3.0
   Compiling utf8_iter v1.0.4
   Compiling tower-service v0.3.3
   Compiling base64 v0.22.1
   Compiling icu_properties_data v2.3.0
   Compiling indexmap v2.14.2
   Compiling httparse v1.10.1
   Compiling fnv v1.0.7
   Compiling rustix v1.1.5
   Compiling utf8parse v0.2.2
   Compiling linux-raw-sys v0.12.1
   Compiling generic-array v0.14.7
   Compiling openssl v0.10.81
   Compiling cpufeatures v0.2.17
   Compiling atomic-waker v1.1.2
   Compiling try-lock v0.2.5
   Compiling lexical-parse-integer v1.0.6
   Compiling lexical-write-integer v1.0.6
   Compiling lazy_static v1.5.1
   Compiling semver v1.0.28
   Compiling crossbeam-epoch v0.9.21
   Compiling foreign-types-shared v0.1.1
   Compiling httpdate v1.0.3
   Compiling tower-layer v0.3.3
   Compiling want v0.3.2
   Compiling anstyle-parse v1.0.0
   Compiling form_urlencoded v1.2.2
   Compiling sync_wrapper v1.0.2
   Compiling lexical-write-float v1.0.6
   Compiling futures-channel v0.3.34
   Compiling lexical-parse-float v1.0.6
   Compiling foreign-types v0.3.2
   Compiling rustc_version v0.4.1
   Compiling anstyle v1.0.14
   Compiling native-tls v0.2.18
   Compiling anstyle-query v1.1.5
   Compiling either v1.18.0
   Compiling crossbeam-deque v0.8.8
   Compiling paste v1.0.15
   Compiling is_terminal_polyfill v1.70.2
   Compiling colorchoice v1.0.5
   Compiling mime v0.3.17
   Compiling proc-macro2-diagnostics v0.10.1
   Compiling http-body v1.1.0
   Compiling jobserver v0.1.35
   Compiling ipnet v2.12.2
   Compiling thiserror v2.0.21
   Compiling rayon-core v1.13.0
   Compiling anstream v1.0.0
   Compiling openssl-probe v0.2.1
   Compiling base64 v0.23.1
   Compiling block-buffer v0.10.4
   Compiling crypto-common v0.1.7
   Compiling http-body-util v0.1.5
   Compiling cfg_aliases v0.2.2
   Compiling crc32fast v1.5.2
   Compiling flatbuffers v24.12.23
   Compiling encoding_rs v0.8.42
   Compiling cc v1.6.0
   Compiling lexical-core v1.0.6
   Compiling digest v0.10.7
   Compiling ucd-trie v0.1.7
   Compiling nix v0.30.1
   Compiling same-file v1.0.6
   Compiling utf-8 v0.7.6
   Compiling timely_container v0.27.0
   Compiling unicode-width v0.2.2
   Compiling bytemuck v1.25.2
   Compiling sha1 v0.10.7
   Compiling winnow v0.7.15
   Compiling adler2 v2.0.1
   Compiling cfg_aliases v0.1.1
   Compiling alloc-no-stdlib v2.0.4
   Compiling toml_write v0.1.2
   Compiling unicase v2.10.0
   Compiling zstd-safe v7.2.1
   Compiling data-encoding v2.11.1
   Compiling jiff-core v0.1.1
   Compiling yansi v1.0.1
   Compiling bitflags v1.3.2
   Compiling simd-adler32 v0.3.10
   Compiling nix v0.28.0
   Compiling regex-automata v0.4.18
   Compiling alloc-stdlib v0.2.4
   Compiling getopts v0.2.24
   Compiling mime_guess v2.0.5
   Compiling num-integer v0.1.47
   Compiling num-complex v0.4.6
   Compiling errno v0.3.14
   Compiling socket2 v0.6.5
   Compiling mio v1.2.4
   Compiling getrandom v0.2.17
   Compiling atoi v2.0.0
   Compiling signal-hook-registry v1.4.8
   Compiling rand_core v0.9.5
   Compiling miniz_oxide v0.9.1
   Compiling timely_logging v0.27.0
   Compiling rand_core v0.6.4
   Compiling pest v2.9.2
   Compiling walkdir v2.5.0
   Compiling num-bigint v0.4.8
   Compiling num-iter v0.1.46
   Compiling anyhow v1.0.104
   Compiling csv-core v0.1.13
   Compiling uncased v0.9.10
   Compiling multiversion_no_op v1.0.0
   Compiling snap v1.1.2
   Compiling timely_bytes v0.27.0
   Compiling core_detect v1.0.0
   Compiling simdutf8 v0.1.5
   Compiling zeroize v1.9.0
   Compiling getrandom v0.4.3
   Compiling rayon v1.12.0
   Compiling csv v1.4.0
   Compiling brotli-decompressor v4.0.3
   Compiling ordered-float v2.10.1
   Compiling flate2 v1.1.10
   Compiling rustls-pki-types v1.15.1
   Compiling xattr v1.6.1
   Compiling jiff v0.2.37
   Compiling num_cpus v1.17.0
   Compiling cpu-time v1.0.0
   Compiling filetime v0.2.29
   Compiling itertools v0.14.0
   Compiling sharded-slab v0.1.7
   Compiling tracing-log v0.2.0
   Compiling combine v4.6.8
   Compiling thread_local v1.1.10
   Compiling figment v0.10.19
   Compiling synstructure v0.14.0
   Compiling openssl-sys v0.9.117
   Compiling zstd-sys v2.0.13+zstd.1.5.6
   Compiling tikv-jemalloc-sys v0.6.1+5.3.0-1-ge13ca993e8ccb9ba9847cc330696e02839f328f7
   Compiling time-core v0.1.9
   Compiling num-conv v0.2.2
   Compiling strsim v0.11.1
   Compiling pest_meta v2.9.2
   Compiling static_assertions v1.1.0
   Compiling num-rational v0.4.2
   Compiling endian-type v0.1.2
   Compiling twox-hash v2.1.5
   Compiling base64ct v1.8.3
   Compiling deranged v0.5.8
   Compiling powerfmt v0.2.1
   Compiling inlinable_string v0.1.15
   Compiling clap_lex v1.1.1
   Compiling integer-encoding v3.0.4
   Compiling nu-ansi-term v0.50.3
   Compiling lz4_flex v0.11.6
   Compiling num v0.4.3
   Compiling thrift v0.17.0
   Compiling clap_builder v4.6.7
   Compiling password-hash v0.5.0
   Compiling twox-hash v1.6.3
   Compiling brotli v7

... [14293 bytes truncated] ...

ing rand v0.8.8
   Compiling url v2.5.8
   Compiling toml v0.8.23
   Compiling arrow-data v53.4.1
   Compiling tungstenite v0.24.0
   Compiling timely v0.27.0
   Compiling hnsw_rs v0.3.4 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/baseline-build/527bacc306ae480504adbdec56c6823672629793-8dc912acd5bf/src/vendor/hnsw_rs)
   Compiling arrow-array v53.4.1
   Compiling tokio-util v0.7.19
   Compiling tower v0.5.3
   Compiling tokio-native-tls v0.3.1
   Compiling tokio-tungstenite v0.24.0
   Compiling arrow-select v53.4.1
   Compiling arrow-arith v53.4.1
   Compiling arrow-row v53.4.1
   Compiling h2 v0.4.19
   Compiling tower-http v0.6.11
   Compiling tower-http v0.5.2
   Compiling differential-dataflow v0.20.0
   Compiling arrow-cast v53.4.1
   Compiling arrow-ord v53.4.1
   Compiling arrow-string v53.4.1
   Compiling zstd v0.13.3
   Compiling arrow-ipc v53.4.1
   Compiling arrow-csv v53.4.1
   Compiling arrow-json v53.4.1
   Compiling hyper v1.11.1
   Compiling parquet v53.4.1
   Compiling arrow v53.4.1
   Compiling hyper-util v0.1.21
   Compiling hyper-tls v0.6.0
   Compiling axum v0.7.9
   Compiling reqwest v0.12.28
   Compiling inputlayer-ontology-client v0.1.0 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/baseline-build/527bacc306ae480504adbdec56c6823672629793-8dc912acd5bf/src/ontology-client)
   Compiling tikv-jemallocator v0.6.1
   Compiling inputlayer v0.1.0 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/baseline-build/527bacc306ae480504adbdec56c6823672629793-8dc912acd5bf/src)
    Finished `release` profile [optimized] target(s) in 2m 56s
=== Benchmark (680f50d1ae50e8f9a57c83634f932bf3e98795d6) ===
  P0_no_subs: ack p50 266.9 ms, write->delta p50/p99 NaN/NaN ms, read p50 0.9 ms, cpu 3.3 ms/write, 0 late
  P1_r1_unbound_x1: ack p50 342.3 ms, write->delta p50/p99 342.9/358.2 ms, read p50 1.0 ms, cpu 3.3 ms/write, 0 late
  P2_r1_unbound_x100: ack p50 173.3 ms, write->delta p50/p99 174.1/245.6 ms, read p50 1.0 ms, cpu 10.0 ms/write, 0 late
  P3_hot_reach_unbound_x1: ack p50 154.5 ms, write->delta p50/p99 157.9/219.9 ms, read p50 1.0 ms, cpu 3.3 ms/write, 0 late
  P4_r1_keyed_x1: ack p50 157.0 ms, write->delta p50/p99 157.4/207.6 ms, read p50 0.9 ms, cpu 3.3 ms/write, 0 late
  P5_hot_reach_keyed_x1: ack p50 205.7 ms, write->delta p50/p99 209.5/214.8 ms, read p50 1.3 ms, cpu 6.7 ms/write, 0 late
  P6_r1_unbound_x1_rules9: ack p50 13.2 ms, write->delta p50/p99 14.1/14.8 ms, read p50 1.2 ms, cpu 3.3 ms/write, 0 late
baseline 1000 edges: loaded in 1296 ms, 8 reads, 7 phases, peak RSS 43 MB
  P0_no_subs: ack p50 4.0 ms, write->delta p50/p99 NaN/NaN ms, read p50 2.6 ms, cpu 6.7 ms/write, 0 late
  P1_r1_unbound_x1: ack p50 19.8 ms, write->delta p50/p99 20.2/42.8 ms, read p50 5.8 ms, cpu 3.3 ms/write, 0 late
  P2_r1_unbound_x100: ack p50 50.9 ms, write->delta p50/p99 56.2/81.2 ms, read p50 13.0 ms, cpu 6.7 ms/write, 0 late
  P3_hot_reach_unbound_x1: ack p50 2.9 ms, write->delta p50/p99 8.4/21.2 ms, read p50 2.0 ms, cpu 6.7 ms/write, 0 late
  P4_r1_keyed_x1: ack p50 4.1 ms, write->delta p50/p99 4.5/14.9 ms, read p50 1.2 ms, cpu 3.3 ms/write, 0 late
  P5_hot_reach_keyed_x1: ack p50 14.9 ms, write->delta p50/p99 21.3/38.6 ms, read p50 1.7 ms, cpu 6.7 ms/write, 0 late
  P6_r1_unbound_x1_rules9: ack p50 6.4 ms, write->delta p50/p99 7.2/9.8 ms, read p50 1.1 ms, cpu 3.3 ms/write, 0 late
candidate 1000 edges: loaded in 53 ms, 8 reads, 7 phases, peak RSS 41 MB
# Views: cost of reads and writes against deployed rules

- baseline: 527bacc306ae480504adbdec56c6823672629793 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/perf-gate/baseline-servers/527bacc306ae480504adbdec56c6823672629793-8dc912acd5bf/inputlayer-server)
- candidate: 680f50d1ae50e8f9a57c83634f932bf3e98795d6 (~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/bench-views/runs/20261006T055359Z/inputlayer-server)
- host computeinstance-e00w59r9a240k9qz1e (32 CPUs, server CPUs any), load average 63.65 50.93 39.50 8/1794 3765484 -> 63.79 52.13 40.36 34/1607 3801793

## 1000 edges (baseline)

Load: 1002 edges, 4 hot heads in 1296 ms (server CPU 10 ms), RSS 25 MB. Peak RSS 43 MB.

Registration: fillers_to_5 2592 ms, hot_reach 1162 ms, path 1130 ms, path_step 1269 ms, r1 792 ms.

| read | rules | rows | p50 / p99 ms | server CPU ms/read | rule evaluations/read | view reads/read |
|---|---|---|---|---|---|---|
| `?edge(400,Y)` | 4 | 1 | 0.4 / 0.8 | 0.0 | 0.0 | 0.0 |
| `?r1(X,Y)` | 4 | 4 | 0.6 / 0.8 | 5.0 | 1.0 | 0.0 |
| `?r1(400,Y)` | 4 | 1 | 0.6 / 0.6 | 0.0 | 1.0 | 0.0 |
| `?path(400,Y)` | 4 | 3 | 2.6 / 3.4 | 0.0 | 1.0 | 0.0 |
| `?hot_reach(X,Y)` | 4 | 12 | 2.9 / 94.0 | 5.0 | 1.0 | 0.0 |
| `?hot_reach(400,Y)` | 4 | 3 | 3.4 / 67.3 | 0.0 | 1.0 | 0.0 |
| `?r1(X,Y)` | 9 | 22 | 1.1 / 1.4 | 0.0 | 1.0 | 0.0 |
| `?r1(400,Y)` | 9 | 5 | 0.8 / 1.0 | 0.0 | 1.0 | 0.0 |

| phase | subs | ack p50 / p99 ms | write->delta p50 / p99 ms | read during p50 ms | server CPU ms/write | frames/write | rule evaluations/write | maintenance us/write | RSS MB | late |
|---|---|---|---|---|---|---|---|---|---|---|
| P0_no_subs | 0 | 266.9 / 282.1 | - | 0.9 | 3.3 | 0.0 | 1.0 | 0.0 | 29 | 0 |
| P1_r1_unbound_x1 | 1 | 342.3 / 357.6 | 342.9 / 358.2 | 1.0 | 3.3 | 2.0 | 2.0 | 0.0 | 29 | 0 |
| P2_r1_unbound_x100 | 100 | 173.3 / 244.6 | 174.1 / 245.6 | 1.0 | 10.0 | 200.0 | 2.0 | 0.0 | 36 | 0 |
| P3_hot_reach_unbound_x1 | 1 | 154.5 / 216.9 | 157.9 / 219.9 | 1.0 | 3.3 | 2.0 | 2.0 | 0.0 | 38 | 0 |
| P4_r1_keyed_x1 | 1 | 157.0 / 207.0 | 157.4 / 207.6 | 0.9 | 3.3 | 2.0 | 2.0 | 0.0 | 38 | 0 |
| P5_hot_reach_keyed_x1 | 1 | 205.7 / 210.2 | 209.5 / 214.8 | 1.3 | 6.7 | 2.0 | 2.0 | 0.0 | 39 | 0 |
| P6_r1_unbound_x1_rules9 | 1 | 13.2 / 14.3 | 14.1 / 14.8 | 1.2 | 3.3 | 2.0 | 2.0 | 0.0 | 38 | 0 |

Subscribe: 1 x `?r1(X,Y)` 2 ms, 100 x `?r1(X,Y)` 97 ms, 1 x `?hot_reach(X,Y)` 66 ms, 1 x `?r1(0,Y)` 3 ms, 1 x `?hot_reach(0,Y)` 35 ms, 1 x `?r1(X,Y)` 2 ms.

## 1000 edges (candidate)

Load: 1002 edges, 4 hot heads in 53 ms (server CPU 10 ms), RSS 23 MB. Peak RSS 41 MB.

Registration: fillers_to_5 826 ms, hot_reach 81 ms, path 27 ms, path_step 39 ms, r1 25 ms.

| read | rules | rows | p50 / p99 ms | server CPU ms/read | rule evaluations/read | view reads/read |
|---|---|---|---|---|---|---|
| `?edge(400,Y)` | 4 | 1 | 0.5 / 0.8 | 0.0 | 0.0 | 0.0 |
| `?r1(X,Y)` | 4 | 4 | 0.9 / 1.2 | 0.0 | 1.0 | 0.0 |
| `?r1(400,Y)` | 4 | 1 | 1.2 / 19.0 | 5.0 | 1.0 | 0.0 |
| `?path(400,Y)` | 4 | 3 | 7.5 / 28.1 | 5.0 | 1.0 | 0.0 |
| `?hot_reach(X,Y)` | 4 | 12 | 12.2 / 22.6 | 5.0 | 1.0 | 0.0 |
| `?hot_reach(400,Y)` | 4 | 3 | 11.8 / 29.8 | 0.0 | 1.0 | 0.0 |
| `?r1(X,Y)` | 9 | 22 | 1.2 / 3.5 | 0.0 | 1.0 | 0.0 |
| `?r1(400,Y)` | 9 | 5 | 1.4 / 4.4 | 5.0 | 1.0 | 0.0 |

| phase | subs | ack p50 / p99 ms | write->delta p50 / p99 ms | read during p50 ms | server CPU ms/write | frames/write | rule evaluations/write | maintenance us/write | RSS MB | late |
|---|---|---|---|---|---|---|---|---|---|---|
| P0_no_subs | 0 | 4.0 / 15.0 | - | 2.6 | 6.7 | 0.0 | 1.0 | 0.0 | 31 | 0 |
| P1_r1_unbound_x1 | 1 | 19.8 / 23.9 | 20.2 / 42.8 | 5.8 | 3.3 | 2.0 | 2.0 | 0.0 | 31 | 0 |
| P2_r1_unbound_x100 | 100 | 50.9 / 79.7 | 56.2 / 81.2 | 13.0 | 6.7 | 200.0 | 2.0 | 0.0 | 37 | 0 |
| P3_hot_reach_unbound_x1 | 1 | 2.9 / 9.4 | 8.4 / 21.2 | 2.0 | 6.7 | 2.0 | 2.0 | 0.0 | 39 | 0 |
| P4_r1_keyed_x1 | 1 | 4.1 / 5.4 | 4.5 / 14.9 | 1.2 | 3.3 | 2.0 | 2.0 | 0.0 | 38 | 0 |
| P5_hot_reach_keyed_x1 | 1 | 14.9 / 29.0 | 21.3 / 38.6 | 1.7 | 6.7 | 2.0 | 2.0 | 0.0 | 38 | 0 |
| P6_r1_unbound_x1_rules9 | 1 | 6.4 / 9.1 | 7.2 / 9.8 | 1.1 | 3.3 | 2.0 | 2.0 | 0.0 | 38 | 0 |

Subscribe: 1 x `?r1(X,Y)` 10 ms, 100 x `?r1(X,Y)` 72 ms, 1 x `?hot_reach(X,Y)` 27 ms, 1 x `?r1(0,Y)` 4 ms, 1 x `?hot_reach(0,Y)` 9 ms, 1 x `?r1(X,Y)` 7 ms.

result file: ~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/bench-views/runs/20261006T055359Z/result.json

Summary: ~/.no-mistakes/worktrees/9eaf1dfa27af/01M47TEA9EZ3HX2FPK9AJ26GTJ/target/bench-views/runs/20261006T055359Z/summary.md

real	6m58.033s
user	34m23.846s
sys	2m56.891s

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info

⚠️ scripts/perf-gate-baseline.sh:36 - The per-baseline work directory (target/perf-gate/baseline-build/<sha>-<toolchain>, an extracted tree plus a full release target dir) is removed only on the success path at line 47. Under set -e, a baseline that fails to compile, or a run that is killed mid-build, leaves it behind; it is reclaimed only if the same commit and toolchain is requested again (line 37). The old shared target/perf-gate/build was bounded at one directory; now a bisect over N commits that do not build leaves N full target directories on the benchmark host. Remedy: set trap 'rm -rf "$WORK"' EXIT once WORK is created, and drop the explicit rm -rf at line 47. The selftest's "no build directory is left behind" check (scripts/perf-gate-baseline-selftest.sh:103) covers only the success path.
⚠️ scripts/perf-gate-baseline.sh:44 - Simplification: the copy-to-$BIN.tmp-then-mv at lines 44-46 is a new component that guards against an interrupted copy being taken for a cached binary. No intent requirement needs it: the intent is that each baseline is built from its own sources, which the per-baseline source and target directory already delivers, and the previous code used a plain cp. Recommend reverting to a single cp into $BIN, unless the author wants to keep this extra hardening.
ℹ️ scripts/perf-gate-baseline.sh:32 - Moving the cache to baseline-servers/ correctly stops any binary built in the shared directory from being read again, but nothing removes the existing target/perf-gate/build, target/perf-gate/src and target/perf-gate/servers/<sha>-* on the benchmark host's clone. They are now dead disk (servers/candidate is still used by scripts/perf-gate.sh:74) and need a one-off manual cleanup. Commits that predate this change still run their own old script when checked out on the host, so they keep using the shared directory; they cannot poison the new cache.

🔧 Fix applied.
1 info still open:

ℹ️ scripts/perf-gate-baseline.sh:32 - Moving the cache to baseline-servers/ correctly stops any binary built in the shared directory from being read again, but nothing removes the existing target/perf-gate/build, target/perf-gate/src and target/perf-gate/servers/<sha>-* on the benchmark host's clone. They are now dead disk (servers/candidate is still used by scripts/perf-gate.sh:74) and need a one-off manual cleanup. Commits that predate this change still run their own old script when checked out on the host, so they keep using the shared directory; they cannot poison the new cache.

✅ **Test** - passed

✅ No issues found.

Live validation: ✅ go - 8 of 8 scenarios driven live against the product

Scenario	Result	Live	Evidence
Build the real #387 baseline, then its older parent, on one clone: each binary is its own commit's server	✅ pass	live	real-newer-then-older.txt and real-binaries-compared.txt: both builds exit 0; `527bacc` binary sha256 4382b65d… with 0 `access_denied` strings, `efa32ce` binary cc89631e… with 10
Same order with the base commit's shared build directory reproduces the bug: the older baseline comes back as the newer server	✅ pass	live	real-old-newer-then-older.txt and real-binaries-compared.txt: the `527bacc` 'build' finishes in 3 s with exit 0 and its binary is byte-identical to #387's (cc89631e…, 10 `access_denied` strings)
Fixture workspace, both orders (newer then older, older then newer): each baseline prints its own engine and protocol version	✅ pass	live	selftest-fixed.txt (all PASS); selftest-old-shared-dir.txt shows the same checks failing on the pre-change build ('engine-newer protocol-newer' for the older baseline, and the reverse)
Asking for an already built baseline returns the cached binary without rebuilding	✅ pass	live	selftest-fixed.txt ('a cached baseline is not built again'); gate-aa-quick.txt has no 'Build baseline server' line for the cached `527bacc`
A baseline that does not compile fails with non-zero status, caches nothing and leaves no build directory	✅ pass	live	selftest-fixed.txt: 'a baseline that does not compile fails', 'no build directory is left behind, built or not'
Interrupting a baseline build mid-flight (SIGINT or SIGTERM) removes its build directory and caches no binary	✅ pass	live	interrupt-cleanup.txt: `baseline-build` holds the commit's directory during the build and is empty after each signal; `baseline-servers` is empty
Run the perf gate end to end against a baseline commit: it measures the helper's binary for that commit	✅ pass	live	gate-aa-quick.txt: exit 0, summary reports baseline `527bacc` with binary sha256 4382b65d6c57, the same binary the helper built for that commit
Run the views benchmark with --baseline-rev: the baseline is built by the helper and measured next to the candidate	✅ pass	live	bench-views-baseline.txt: exit 0, 'Build baseline server `527bacc`…', baseline path under `target/perf-gate/baseline-servers/`, both arms measured

scripts/perf-gate-baseline-selftest.sh (fixed helper: all 8 checks pass)
scripts/perf-gate-baseline-selftest.sh in a scratch repo whose scripts/perf-gate-baseline.sh is the base commit's shared-target/perf-gate/build block (3 checks fail: reproduces the bug)
scripts/perf-gate-baseline.sh efa32cec then scripts/perf-gate-baseline.sh 527bacc3 on the real repo (170 s and 189 s full release builds, both exit 0, baseline-build/ empty after each)
old-shared-dir-baseline-shim.sh efa32cec then 527bacc3 on the real repo (second 'build' finishes in 3 s and returns the #387 binary)
sha256sum and grep -c -a access_denied over the four resulting inputlayer-server binaries
interrupt-check.sh scripts/perf-gate-baseline.sh: timeout -s INT 8 and timeout -s TERM 8 on a baseline build held open by a sleeping build script
scripts/perf-gate.sh --aa --baseline-rev 527bacc3 --profile quick --rounds 1 --fixtures cheap_query --no-verdict (exit 0)
scripts/bench-views.sh --baseline-rev 527bacc3 --edges 1000 --writes 3 --large-writes 2 --reads 2 --keyed 1 --recursive-keyed 1 --large-recursive-keyed 1 --filler-rules 5 (exit 0)

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

scripts/perf-gate.sh, bench-sessions.sh and bench-views.sh extracted each baseline with `git archive` and built all of them in one shared target/perf-gate/build. Cargo names a workspace crate's artifacts by its path inside the workspace, not by where the workspace is, and rebuilds one only when a source file is newer than the last build. `git archive` dates every file at its commit, which is older than any build already in the directory, so a later baseline reused an earlier one's workspace crates: the build failed where the APIs had diverged (`missing field code in ServerFrame` after #387), linked another commit's `ws-protocol` or `ontology-client` where they had not, and returned the earlier baseline's whole server where no file had been added or removed between the two commits. The three scripts now build baselines through scripts/perf-gate-baseline.sh: each baseline gets a source and target directory of its own under target/perf-gate/baseline-build/, removed once the binary is cached. The cache moves to target/perf-gate/baseline-servers/, so a binary cached by the shared build is never read again, and the three scripts now share one toolchain key (the gate hashed `rustc -V` without its newline, the benchmarks with it). scripts/perf-gate-baseline-selftest.sh builds two baselines that differ in the root crate and in a path dependency, in both orders, and checks each binary reports its own commit. It runs in `make pre-pr-selftest` and in the full suite's perf-gate tool job.

… on failure too

jsam · 2026-10-06T06:37:38Z

Moved to https://github.com/inputlayer/inputlayer-symbolic/pull/3

jsam added 3 commits October 6, 2026 05:18

no-mistakes(review): clean baseline build dir on failure; plain cp

680f50d

no-mistakes(document): docs(perf-gate): baseline build dir is removed…

5038132

… on failure too

jsam closed this Oct 6, 2026

Navigation Menu

Repository navigation

fix(perf-gate): build every baseline in a target directory of its own - #399

fix(perf-gate): build every baseline in a target directory of its own#399
jsam wants to merge 3 commits into
mainfrom
fm/il-perf-gate-stale-baseline

jsam commented Oct 6, 2026

Uh oh!

jsam commented Oct 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

jsam commented Oct 6, 2026

Intent

What Changed

Risk Assessment

Testing

Pipeline

Uh oh!

jsam commented Oct 6, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant