Repository navigation
test(scenarios): scenario harness on the testkit, with the reactive e2e suite moved into it - #358
Merged
Merged
Conversation
jsam
changed the title
This was referenced Oct 5, 2026
jsam
changed the title
jsam
changed the title
…2e suite into it Move tests/e2e_reactive into one tests/scenarios binary that runs under plain cargo test (debug); make e2e-reactive stays the release run that writes latency samples, and make lint covers the binary through --all-targets. Testkit additions: EngineBuilder::views(Mode) with Mode::from_env (maintained is refused with a clear panic until V2 #309 defines engine.views), memory_limits, max_query_cost and nesting_limit settings; Engine::metrics() reading /metrics/prometheus into Counters with Option fields and Violation::NotMeasurable for a missing counter; Engine::create_user, grant, create_api_key and WsClient::connect_with_key; WsClient::execute_expecting and execute_at with the reply revision; Fixture::shop_pack(Size) with the shop rules (near only in Size::Vector). Fixture::install commits consecutive statements as one program. A harness module checks these against a real engine. TESTING.md documents the suite and the harness. Refs #335
…se-only Rebased onto main, which added tests/e2e_reactive/saturation.rs while e2e_reactive was test = false. Its delivery bounds need a release engine (in debug a probe delta took 8 s against the 5 s bound), so it is ignored in debug builds and runs in make e2e-reactive.
jsam
changed the title
jsam
marked this pull request as ready for review
October 5, 2026 20:10
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
make sure that all those new changes we are making in accordance with the production readiness are properly covered by e2e / integration testing cause all of them are related so they cannot be tested in isolation only ... we have to check them through scenarios which we have to craft ... please ensure that all those different testing we have is cleaned up and that we bring some structure to it so we dont have too much redudancy ... our testing is getting huge and slow so ensure that we are always testing the right stuff and that this is all properly documented in TESTING.md
(Context: implement issue #335 exactly as written, part of the test strategy in ~/firstmate/data/test-strategy/report.md and TESTING.md draft.)
Issue #335 "Testing: Scenario harness (T-01)", as written:
Scope.
tests/scenarios/main.rs(test = true, debug-capable) built oninputlayer-testkit. Movetests/e2e_reactive/*in asscenarios::reactive,scenarios::stream,scenarios::delivery,scenarios::wireunchanged; keepmake e2e-reactiveas the release run that writes latency samples. Testkit additions:EngineBuilder::views(Mode)writingengine.viewsonce V2 defines it (until then the builder accepts the mode and themaintainedvalue is refused with a clear panic),Mode::from_env("INPUTLAYER_SCENARIO_VIEWS");memory_limits,max_query_cost,nesting_limitsettings;Engine::metrics()->Countersfrom/metrics/prometheuswithOptionfields;Engine::create_user,grant,WsClient::connect_with_key;WsClient::execute_expecting(program, revision, scope)andexecute_at(sendsatand readsrevisionfrom the reply; bothOptionuntil V9/V13);Fixture::shop_pack(size)with the rules in report Section 7.2 (thenearrule only inSize::Vector).The report Section 7.2 shop pack: one fixture so scenarios exercise the related changes together. Base relations order(id, cust, total), order_item(order, item), stock(item, qty), blocked(item), link(item, other), embedding(item, vec), claim(order, item, agent). Deployed rules: eligible(Order, Item, Why) <- order_item(Order, Item), stock(Item, Q), Q > 0, !blocked(Item), Why = "in_stock"; related(Item, Other) <- link(Item, Other); related(Item, Other) <- related(Item, Mid), link(Mid, Other); offer(Order, Other) <- eligible(Order, Item, _), related(Item, Other), !blocked(Other), !claim(Order, Other, _); n_eligible(Order, N) <- eligible(Order, Item, _), N = count(Item); near(Item, Other, D) <- embedding(Item, V), hnsw_nearest("emb_idx", V, 5, Other, D). Order is the agent's key; sizes small (hundreds of rows, every PR) and lab (1M edges, bench host only). Counters the scenarios assert on: rule_evaluations, view_reads, subscription_evaluations, view_maintenance_us; until V1 exports them the helper returns None and an assertion on it is an expected failure (not measurable), never a skip.
Acceptance.
cargo test --test scenariosruns the 14 moved scenarios green in debug in under 60 s on 4 cores;INPUTLAYER_SCENARIO_VIEWS=maintained cargo test --test scenariosfails fast with the "mode not available" message until V2;make lintcovers the binary;shop_pack(Size::Small)installs in under 200 ms;Counters::rule_evaluationsisNoneon this engine and the helper's unit test shows a parsed sample.What Changed
e2e_reactivetest target withtests/scenarios(test = true, so plaincargo testnow runs it in debug). Thereactive,stream,delivery,wireandsaturationmodules moved over;saturationruns only in release. A newharnessmodule checks the testkit pieces against a running engine. Theshop_packvector scenario is#[ignore]d because of the.index createhang (Rare hang: .index create (HNSW, cosine) never replies under load #377).inputlayer-testkit:ModeandEngineBuilder::views:Mode::from_env("INPUTLAYER_SCENARIO_VIEWS")picks the mode, andmaintainedpanics with a clear message until V2.EngineBuildersettings:memory_limits,max_query_costandnesting_limit.Enginemethods:metrics()returnsCountersread from/metrics/prometheuswithOptionfields, pluscreate_userandgrant.WsClientmethods:connect_with_key,execute_expectingandexecute_at.Fixture::shop_pack(Size), where thenearrule exists only inSize::Vector.Violation::NotMeasurablevariant.TESTING.mdupdates:make e2e-reactiveruns--test scenariosin release with--include-ignoredand still writes latency samples.test-allandci-test-allleave the scenarios binary out of theircargo testpass, so the suite runs only once.e2e_reactiveclippy line is gone;make lintnow covers the binary through--all-targets.TESTING.mddocuments the new structure.Risk Assessment
✅ Low: The change only touches test code, the testkit, the Makefile and docs. The uncertified fixer commits are correct and stay within what was decided: the fixture error now reports the program's index, byte size and the engine's errors without the program text, and only the vector scenario is quarantined, with an unconditional #[ignore], --include-ignored in make e2e-reactive, and a TESTING.md note. I found no reachable defect.
Testing
I drove each scenario against real engine processes. The debug suite, pinned to 4 cores, passed 18 scenarios in 26 s with the vector and saturation scenarios ignored. Maintained mode failed all 18 non-ignored scenarios within 0.3 s with the V2 "mode not available" message. The counter unit tests passed.
make e2e-reactivepassed all 20 scenarios in release, including the quarantined vector scenario and saturation, wrote three latency sample files, and printed the shop_pack(Small) install time as 66.7 ms. A no-run build of thetest-alldebug-stage targets listed 82 test executables and none was scenarios. I did not runmake lintbecause this phase may not run linters. Logs are in the evidence directory.cargo test --test scenariosin debug on 4 cores: the 14 moved scenarios and the harness scenarios pass in under 60 scargo test, the quarantined vector scenario is reported as ignored with its #377 reason instead of runningmake e2e-reactiveruns every scenario in release, including the quarantined vector scenario and saturation, and writes latency samplesmake lintcovers the scenarios binaryEvidence: Debug scenarios on 4 pinned cores (18 passed, 2 ignored, 26 s wall)
Evidence: Maintained views mode fails fast with the V2 'mode not available' message
Evidence: Counters unit tests (parsed sample, rule_evaluations None)
Evidence: make e2e-reactive release run (20 passed incl. quarantined vector scenario; latency samples written)
Evidence: test-all debug stage excludes the scenarios binary
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 1 issue found → auto-fixed ✅
TESTING.md- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Review** - 1 issue found → auto-fixed ✅
testkit/src/fixture.rs:73- SinceFixture::installnow groups statements into programs of up to PROGRAM_BYTES (256 KiB), a failed fixture commit produces a huge error.WsClient::commit(testkit/src/client.rs:469) formats the whole program with{program:?}, and install wraps that infixture '{}' statement failed: {e}. So one bad fact in aSize::LaborSize::Smallpack prints up to about 256 KiB of escaped IQL instead of the failing statement. Correctness is unaffected; only the diagnostic suffers. Fix: in install'smap_err, report the program's index and byte length plus the engine's errors rather than relying on the commit message that embeds the program, or truncate the program in the message.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ No issues found.
🔧 **Test** - 1 issue found → auto-fixed ✅
tests/scenarios/harness.rs:66- This scenario failed once in 9 full debug runs on 4 pinned cores:harness::shop_pack_vector_serves_the_near_rulestopped atfixture 'shop_pack_vector' program 1 (53 bytes) failed: timed out waiting for a reply. The engine never answered.index create emb_idx on embedding(vec) metric cosinewithin the testkit's 60 s FRAME_TIMEOUT, which also pushed that run to 62 s, over the 60 s acceptance budget. The hang did not come back when the test ran alone, in 16 parallel copies, or in 120 overloaded harness runs. That points to a rare engine-side hang in index creation, not a harness bug. Since this test now runs on every PR, it may show up as an intermittent PR failure. Decide whether to investigate the engine's index-create path or track it as a known flake.cargo test --test scenariosin debug on 4 cores and the 14 moved scenarios plus the harness scenarios pass in under 60 smake e2e-reactivewith samples sent to the evidence dir, but the cold release build plus the 30 s saturation load had not finished when the structured result was required. Re-run `make e2e…taskset -c 0-3 cargo test --test scenarios -- --nocapture(first run: 18 passed, 1 failed with the vector index-create timeout, 61.9 s)taskset -c 0-3 cargo test --test scenariosrun 8 more times: all 19 passed, 1 ignored (saturation, release-only), 25–28 s walltaskset -c 0-3 cargo test --test scenarios harness::shop_pack_vectoralone, and 16 parallel copies under taskset: all passed120 parallelharness::runs on 4 cores to try to reproduce the index-create hang: no reply timeouts (11 failures were engine-startup credential reads under 40x overload; that code already exists on the base commit)INPUTLAYER_SCENARIO_VIEWS=maintained cargo test --test scenarios: 19 scenarios failed in 0.27 s, each with the 'mode not available' messageINPUTLAYER_SCENARIO_VIEWS=bogus cargo test --test scenarios wire: panics with 'expected "recompute" or "maintained"';=recomputepassescargo test -p inputlayer-testkit: 16 unit tests, including the Counters parse/delta tests and the maintained-mode should_panic testmake --evalexpansion of NON_SCENARIO_TESTS: 73 of the 74 test targets, scenarios left out;cargo test -p inputlayer --all-features --lib --bins <those> --no-runresolves them all and builds no scenarios binarymake -n lint:cargo clippy --workspace --all-targets --all-features -- -D warnings(covers the scenarios binary)make e2e-reactive E2E_REACTIVE_SAMPLES=<evidence dir>: started in the background, not finished when this result was due🔧 Fix applied.
✅ Re-checked - no issues remain.
cargo test --test scenariosin debug on 4 cores is green in under 60 s and skips the quarantined vector scenario with its #377 reasonmake e2e-reactive(release, --include-ignored) runs the quarantined vector scenario and the saturation scenario and writes latency samplestaskset -c 0-3 cargo test --test scenarios(debug, 4 pinned cores)INPUTLAYER_SCENARIO_VIEWS=maintained cargo test --test scenarioscargo test --test scenarios -- --include-ignored harness::shop_pack(debug)make e2e-reactive E2E_REACTIVE_SAMPLES=<evidence dir>(release testkit tests + scenarios with --nocapture --include-ignored)✅ No issues found.
cargo test --test scenariosin debug on 4 cores: the 14 moved scenarios and the harness scenarios pass in under 60 scargo test, the quarantined vector scenario is reported as ignored with its #377 reason instead of runningmake e2e-reactiveruns every scenario in release, including the quarantined vector scenario and saturation, and writes latency samplesmake lintcovers the scenarios binarytaskset -c 0-3 /usr/bin/time -v cargo test --test scenarios(debug, 4 pinned cores)INPUTLAYER_SCENARIO_VIEWS=maintained cargo test --test scenarioscargo test -p inputlayer-testkit counters -- --nocapturemake e2e-reactive(release testkit unit tests +cargo test --release --test scenarios -- --nocapture --include-ignored+ latency sample check)make --evalexpansion of NON_SCENARIO_TESTS,cargo test -p inputlayer --all-features --lib --bins $(NON_SCENARIO_TESTS) --no-runexecutable list, andmake -n test-alldebug-stage command✅ **Document** - passed
✅ No issues found.
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.