Repository navigation
feat(perf-gate): add views benchmark and view/rule-evaluation counters on /metrics/prometheus - #381
Merged
Merged
Conversation
jsam
force-pushed
the
fm/m9-v1-benches-v2
branch
from
October 5, 2026 20:46
48998e4 to
41d343d
Compare
jsam
changed the title
jsam
force-pushed
the
fm/m9-v1-benches-v2
branch
from
October 5, 2026 21:41
41d343d to
4e30aa7
Compare
…e time (#308) Every read of a persistent rule (a query, a standing query's refresh, a proof, a guarded write) whose dependency closure holds a deployed rule now counts one rule evaluation; reads served from a view and view maintenance time get their counters too, at zero until maintained views land. All three are exported on /metrics/prometheus as inputlayer_rule_evaluations_total, inputlayer_view_reads_total and inputlayer_view_maintenance_us_total.
…yed rules (#308) perf-gate views runs the incremental-views review's Section 4 matrix: 10K/100K/1M edges, idle reads of a non-recursive and a recursive view bound and unbound, and closed-loop write phases with 0, 1 and 100 unbound subscribers, 1/10/100/1,000 keyed subscribers, keyed subscribers of the recursive view and a 200-rule catalog. Each phase reports ack, write->delta and the ad-hoc read during writes (p50/p99), server CPU per write, frames per write, RSS, and the engine's rule evaluations per read and per write from /metrics/prometheus. make bench-views and make bench-views-remote (perf-gate-remote.sh --bench views, under the benchmark host's shared lock) run it; the full suite runs it report-only as the Views benchmark job alongside the perf gate.
#308) The incremental-views review's differential-dataflow program (report Appendix A.7) as a Criterion bench: the views benchmark's graph and three rules kept as arrangements on one worker; one insert, retraction and non-hot insert propagated through every view, and a key lookup, at 10K, 100K and 1M edges (VIEW_BENCH_EDGES).
…rix and shared counts
jsam
force-pushed
the
fm/m9-v1-benches-v2
branch
from
October 5, 2026 21:59
4e30aa7 to
c103244
Compare
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
please leave this query batching functionliaty just refactor things so its clear and optimized for this ... auto materialize needs to be default mode of operation where rules get deployed as views and everything else is a query .. make the docs true with the correct systems design ... this is not accceptable state for production readiness (Context: implement #308 exactly as written.)
Issue #308 as written, "Baseline benches and counters for the view work (V1)", part of epic #305 (persistent rules are incrementally maintained views; everything else is a query):
Problem. No bench measures the cost of a write vs graph size, rule count and subscriber count, and no counter tells whether a read of a deployed rule evaluated it. The review's numbers (report Section 4) came from an ad-hoc harness.
Target. (a)
perf-gate views: the Section 4 matrix (10k/100k/1M edges; unbound/keyed subscribers 1/10/100/1,000; recursive and non-recursive views; 200-rule catalog) as a perf-gate scenario with p50/p99 write->delta, ack, ad-hoc read during writes, server CPU per write and RSS; (b) a criterion bench of the maintained-view micro-benchmark (report A.7) as the headroom reference; (c) countersview_reads,rule_evaluations,view_maintenance_usexported on/metrics/prometheus.Acceptance.
make bench-views-remoteruns under the shared lock and reproduces the Section 4 tables within noise;rule_evaluationsincrements once per deployed-rule read today (so V9 can assert 0); the bench is wired intofull-suite.ymlas perf-gate-only.Size M. Dependencies none. Order slot 1.
What Changed
src/execution/view_counters.rs) and exports them on/metrics/prometheusasinputlayer_rule_evaluations_total,inputlayer_view_reads_totalandinputlayer_view_maintenance_us_total. The snapshot's combined-program and plan-cache path tracks whether a program's dependency closure includes a persistent rule. Each such read counts one rule evaluation. A shared standing query or family round also counts once, as does a family cost probe. The provenance baseline re-run uses an uncounted path so it is not counted twice. The REST API and query-profiling guides document the new counters.perf-gate viewsscenario (perf-gate/src/views/mod.rs). It runs a matrix over edge counts, unbound and keyed subscriber counts, recursive and non-recursive views and a 200-rule catalog. It reports p50/p99 write->delta, ack latency, ad-hoc read latency during writes, server CPU per write, RSS and the view counters. You run it withscripts/bench-views.sh,make bench-views, ormake bench-views-remote(on the benchmark host under the shared lock). Infull-suite.yml, aviews-benchjob runs it only onperf_gate_onlydispatches.benches/view_maintenance_benchmarks.rs) for the maintained-view micro-benchmark, as the headroom reference. Also adds aviewsscenario test asserting thatrule_evaluationsgoes up for each read of a deployed rule. The perf-gate README and TESTING.md are updated to match.Risk Assessment
✅ Low: The production change is small: one relaxed atomic increment per evaluation that depends on persistent rules, plus three exported counters. The provenance-baseline double count is fixed through an uncounted variant. Everything else is benchmark tooling, docs and tests, and the views CI job is gated on perf_gate_only as the user decided.
Testing
I built the release server and perf-gate and ran a reducedscripts/bench-views.shmatrix end to end. It exited 0 and wrote summary.md and result.json with ack, write->delta, read-during-writes, CPU per write, RSS, frames and counter columns, and noted the keyed x100 skips (34 hot heads). Against a disposable live server, the metrics endpoint returned 401 without a key and exported the three counters with the admin key. Per-read scrapes showed exactly 1 rule evaluation per deployed-rule read (bound or unbound, recursive or not, and a two-rule join), 0 for a base-relation read, and view_reads and view_maintenance_us stayed at 0. The views scenario tests passed against the real server binary: subscribe and each refresh count 1, and a dirty-session read counts 1. The harness counters test, which used to be a known defect, now passes. The criterion view_maintenance bench ran at 10k–1M edges. Parsing full-suite.yml showed views-bench is gated oninputs.perf_gate_only, which is never set on push, PR or publish (workflow_call) runs. That was a static check only, since GitHub Actions can't be dispatched from this phase, so the scenario is untested. As the user decided, the benchmark-host make target was only dry-run here; its evidence is the earlier remote runs 20261005T185930Z and 20261005T192452Z. I removed the transient bench and criterion outputs from target/. No UI surface changed, so there are no screenshots.make bench-views-remoteruns under the benchmark host's shared lock and reproduces the Section 4 tables within noiseEvidence: bench-views smoke run log (build + run, exit=0)
Evidence: perf-gate views summary.md
Evidence: perf-gate views result.json
Evidence: Prometheus scrape (401 without key; counters with admin key)
Evidence: Per-read counter deltas against live server
?edge(0, X) rows=1 rule_evaluations=0 view_reads=0 ?hop2(0, Z) rows=1 rule_evaluations=1 view_reads=0 ?hop2(X, Z) rows=3 rule_evaluations=1 view_reads=0 ?reach(0, X) rows=4 rule_evaluations=1 view_reads=0 ?reach(X, Y) rows=10 rule_evaluations=1 view_reads=0 ?hop2(0, Z), reach(Z, W) rows=2 rule_evaluations=1 view_reads=0Evidence: views scenario tests output
Evidence: criterion view_maintenance bench output
Evidence: full-suite.yml parsed job gating
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 1 issue found → auto-fixed ✅
src/protocol/rest/handlers/admin.rs- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Review** - 3 issues found → auto-fixed (3) ✅
.github/workflows/full-suite.yml:426- The newviews-benchjob has noif:condition, so it runs on every full-suite run, not only onperf_gate_onlydispatches. That includes release/** pushes and v* tags called from publish.yml, where every job must pass before publish. The run fails with exit 1 on any late delta (over 20 s), any failed subscription or any failed step. The 1M-edge phases are already slow (Regression: keyed rule-view sharing, a write costs 11.9 s CPU and a 643 ms ack at 1M edges / 1,000 keys (was 0.18 s / 3.8 ms) #378: a write costs about 11.9 s CPU with 1,000 keyed subscribers), and they run on a shared CI runner with a 90-minute timeout. So a benchmark described as 'report-only' can block a release. The intent says the bench is 'wired into full-suite.yml as perf-gate-only'. That could mean 'runs only when perf_gate_only is set' or 'report-only like the perf gate'. Please confirm which is meant. If the first: addif: ${{ inputs.perf_gate_only }}(or keep it off the release path) and fix the matching comments at full-suite.yml:8-10, 25 and perf-gate/README.md ('Full-suite runs include it as the Views benchmark job'). If the second: considercontinue-on-error: trueso latency or flakiness cannot block publish.perf-gate/src/views/mod.rs:711- Intent (Baseline benches and counters for the view work (V1) #308) says: 'the Section 4 matrix (10k/100k/1M edges; unbound/keyed subscribers 1/10/100/1,000; ...)'. The scenario does not cover that full grid. (1) Unbound subscribers run only at 1 and 100: P1_r1_unbound_x1, P2_r1_unbound_x100 and P3 at x1 (views/mod.rs:711-714). There is no unbound 10 or 1,000. (2) Keyed counts above the number of hot heads are skipped with a note (views/mod.rs:731-736). With 1% hot heads, 10k edges has only 34 heads, so keyed 100 and 1,000 are skipped there, and 100k edges has 334, so keyed 1,000 is skipped there. The full keyed 1/10/100/1,000 set runs only at 1M edges. (3) Recursive keyed subscribers run only at 1/100, or 1/10 from 1M edges. This may match the review report's actual Section 4 tables, which are not in the repo, but it does not match the literal grid in the issue. Please confirm the reduced grid is intended, or add unbound x10/x1000 phases.docs/content/docs/guides/query-profiling.mdx:176- The docs say each subscription refresh counts as one rule evaluation: 'Reads include ... a subscription's initial result and each refresh'. perf-gate/README.md adds 'P4 with 10 keyed subscribers shows 11 per write'. But keyed subscriptions that differ only in a lifted constant form a parameterized Family (src/protocol/subscription/parameterized.rs). Once the views' own evaluations no longer fit on the compute permits, a family runs one shared round per revision. The counter is recorded per snapshot evaluation (snapshot.rs:431 / 609), so N shared refreshes count as one evaluation, plus occasional probes, not N. P4 at 100 and 1,000 keyed subscribers, and P4 at 10 on a host with fewer than 10 permits, will therefore show far fewer evaluations per write than the docs imply. The counter itself is right (it counts evaluations); the docs are wrong. Fix the docs: say that standing queries sharing a parameterized family count one evaluation per shared round, and qualify the README's '11 per write' example as holding only while the family does not share.🔧 Fix applied.
✅ Re-checked - no issues remain.
docs/content/docs/guides/query-profiling.mdx:176- The docs (and the metric HELP text in src/execution/view_counters.rs:383, 'Reads of persistent rules answered by evaluating…') say rule_evaluations counts reads: queries, a subscription's initial result and each refresh,.why, and conditional writes. A parameterized family's probe also goes through run_query → query_snapshot → execute_cached, so it counts as one evaluation (src/protocol/subscription/parameterized.rs:620-677, 815). No subscriber waits for a probe. So P4 at 100 or 1,000 keyed subscribers can show occasional extra evaluations per write that no read explains. The round-1 finding noted 'plus occasional probes', but that round's doc fix covered only shared rounds. Suggested fix: add one sentence to query-profiling.mdx (and to the perf-gate/README.md 'View work' bullet) saying that a family's probe evaluation also counts once. Keep the counter code as is, per the earlier decision.🔧 Fix applied.
✅ Re-checked - no issues remain.
perf-gate/src/views/mod.rs:839- The write->delta time can never be shorter than the ack.phaseseedslast = Some(reply.at)and then takes the max with each waiter's arrival. So when a subscriber's delta arrives before the writer's reply, the reported write->delta is the ack time, not the time the delta actually arrived. The field is documented as 'Write sent to the last delta it must cause'. Today a refresh re-evaluates the rule after commit, so deltas almost always arrive after the ack and the numbers are right. But the bench exists to compare the later view work (V2/V9), where a maintained view can push its delta before the ack. There, p50/p99 write->delta would be inflated up to the ack without any warning. Fix: startlastfrom the waiters' arrivals only, e.g. fold arrivals into an Option and mark the write late if any waiter returns None, and keepreply.atout of the delta sample.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ No issues found.
🔧 **Test** - 2 issues found → auto-fixed ✅
perf-gate viewswas exercised. To validate, re-run this test phase and let the build finish:scripts/bench-views.sh --edges 10000 --writes 3 --reads 2 --keyed 1,10,100 --filler-rules 50andcargo test --test e2e_reactive views.scripts/bench-views.shwith a reduced matrix runs perf-gate views end-to-end and writes summary.md and result.json with p50/p99 delta, ack, ad-hoc read, CPU per write, RSS and counters; keyed counts above the hot heads are skipped with a note, The view_maintenance_benchmarks criterion bench runs as the headroom reference, views-bench runs only on perf_gate_only dispatches and never on release or tag runs,make bench-views-remoteruns under the benchmark host's shared lockscripts/bench-views.shwith a reduced matrix runs perf-gate views end-to-end and writes summary.md and result.json with p50/p99 delta, ack, ad-hoc read, CPU per write, RSS and counters; keyed counts…make bench-views-remoteruns under the benchmark host's shared lockcargo build --release --manifest-path perf-gate/Cargo.toml --target-dir target && cargo build --release --all-features --bin inputlayer-server(started in the background; had not finished when the result was due)python3 yaml.safe_load(.github/workflows/full-suite.yml): views-bench hasif: ${{ inputs.perf_gate_only }}andneeds: gate; every non-perf job has!inputs.perf_gate_only;workflow_calldeclares no inputs and publish.yml calls full-suite with none🔧 Fix applied.
✅ Re-checked - no issues remain.
cargo test --test e2e_reactive views::every_read_of_a_deployed_rule_evaluates_it_oncescripts/bench-views.shwith a reduced matrix runs perf-gate views end to end and writes summary.md and result.json with p50/p99 write->delta, ack, read during writes, CPU per write, RSS and counter…make bench-views-remoteruns under the benchmark host's shared lock and reproduces the Section 4 tables within noisesystemd-run --user --scope -p MemoryMax=16Gwith CARGO_BUILD_JOBS=4: built perf-gate and inputlayer-server in release in the foreground (both already current)scripts/bench-views.sh --edges 10000 --writes 3 --reads 2 --keyed 1,10,100 --filler-rules 50: exit 0; checked summary.md and result.jsoncargo test --test e2e_reactive views: 3 passedReverted the handler.rs fix temporarily and rancargo test --test e2e_reactive views::a_read_on_a_session: it FAILED with rule_evaluations 2 vs 1; restored the fixLive release server in a temp data dir: logged in as admin, created a key with.apikey create, scraped /metrics/prometheus with and without the key, and took per-read counter deltas for edge/hop2/reach reads, bound and unboundLive.subscribe s ?hop2(X,Z): counter deltas for the initial result, a relevant commit and an irrelevant commitParsed full-suite.yml and publish.yml with PyYAML: views-benchif, perf_gate_only input, how publish.yml calls full-suiteRead the Makefile bench-views-remote target and the lock handling in perf-gate-remote.shcargo bench --bench view_maintenance_benchmarks -- --warm-up-time 1 --measurement-time 2 --sample-size 10✅ No issues found.
cargo test --test scenarios views-> scenarios-views-tests.log (3 passed); smoke result shows P1 rule evaluations/write 2.0 = 1 read during the write + 1 refreshmake -n bench-views-remoteexpands to ./scripts/perf-gate-remote.sh --rev HEAD --bench views, and t…CARGO_BUILD_JOBS=4 systemd-run --user --scope -p MemoryMax=16G cargo build --releasefor perf-gate and inputlayer-server (all features), run in the foregroundscripts/bench-views.sh --edges 10000 --writes 3 --reads 2 --keyed 1,10,100 --filler-rules 50(exit 0; summary.md and result.json inspected)cargo test --test scenarios views(3 e2e tests: reads of deployed rules count once, a session with session facts counts once, a subscription counts once per relevant commit)Manual: started the release inputlayer-server in a temp data dir, logged in as admin, created a KG and an API key, deployed r1/path rules, then scraped/metrics/prometheusaround each read (node script metrics-scrape.mjs)cargo bench --bench view_maintenance_benchmarks -- --warm-up-time 1 --measurement-time 2 --sample-size 10Parsed.github/workflows/full-suite.ymlwith PyYAML to check the views-benchif:and the perf_gate_only input;make -n bench-views-remote; read the flock logic in scripts/perf-gate-remote.sh✅ No issues found.
cargo test --test scenarios -- views:: harness::counters_scrape_the_running_engine(4 passed: one evaluation per deployed-rule read, per subscription refresh, per dirty-session read; required counter scrape)cargo test --manifest-path perf-gate/Cargo.toml --target-dir target views::(incl. write_to_delta_is_the_last_arrival_and_not_floored_at_the_ack)CARGO_BUILD_JOBS=8 systemd-run --user --scope -p MemoryMax=24G cargo build --release ...(perf-gate + inputlayer-server --all-features)scripts/bench-views.sh --edges 10000 --writes 3 --reads 2 --keyed 1,10,100 --filler-rules 50(exit 0)Manual: release inputlayer-server on port 18731 with a temp data dir; Node WS client logs in as admin, creates an API key and a KG, deploys hop/reach rules, scrapes /metrics/prometheus with no key, then with the admin key, before and after each readcargo bench --bench view_maintenance_benchmarks -- --testParsed full-suite.yml and publish.yml with PyYAML to check the views-benchif: inputs.perf_gate_onlygating and that publish.yml calls full-suite with no inputsmake -n bench-views-remote REV=HEAD(resolves to scripts/perf-gate-remote.sh --bench views, which takes the shared flock)✅ No issues found.
make bench-views-remoteruns under the benchmark host's shared lock and reproduces the Section 4 tables within noiseCARGO_BUILD_JOBS=4 systemd-run --user --scope -p MemoryMax=16G scripts/bench-views.sh --edges 10000 --writes 3 --reads 2 --keyed 1,10,100 --filler-rules 50(release build + live perf-gate views run, exit 0)Manual: started the releaseinputlayer-serveron a throwaway data dir (port 18931). Scraped/metrics/prometheuswithout a key (401) and with the generated admin key, then drove reads over WebSocket with a Node script and took a counter scrape before and after each readcargo test --test scenarios -- views:: counters_scrape_the_running_engine(4 passed; each starts the real server binary)cargo bench --bench view_maintenance_benchmarks -- --warm-up-time 1 --measurement-time 2 --sample-size 10Parsed .github/workflows/full-suite.yml with PyYAML: the jobs'if:conditions, the perf_gate_only input, and that workflow_call has no inputs (publish.yml calls it without perf_gate_only)make -n bench-views-remote REV=HEAD(dry run only: expands toscripts/perf-gate-remote.sh --rev HEAD --bench views --)✅ **Document** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.
✅ No issues found.