Repository navigation
test(scenarios): S2-S6, S8 and S17 as expected failures naming their issues - #371
Merged
Merged
Conversation
…issues (#337) The milestone 9 scenarios on the shop pack, each running the part that holds today as a required pass and asserting the rest of the target contract as a deterministic expected failure that names the issue making it hold: S2 and S3 (reads evaluate no rule: V1 #308 counters, then V9 V13 #316), S5 (key isolation for 200 keyed and 10 unkeyed agents; one maintenance pass V1 #308, then V10 #317), S6 (`.subscribe` of a query refused, V11 #318), S8 (exact old-vs-new delta on a rule replacement; drop refusal V7 #314) and S17 (read-your-writes through a rebuild; `views_at` V14 #320). The testkit gains `Violation::UnexpectedWork`, `KnownDefect::judge_first` for checks that run into one issue after another, `try_execute_at`, `QueryResult::views_at`, and an XFAIL log (`INPUTLAYER_XFAIL_LOG`) that `make unit-test`, `make ci-test-all` and `make e2e-reactive` print as the open expected-failure list when the run ends.
jsam
changed the title
jsam
marked this pull request as ready for review
October 5, 2026 21:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
please leave this query batching functionliaty just refactor things so its clear and optimized for this ... auto materialize needs to be default mode of operation where rules get deployed as views and everything else is a query .. make the docs true with the correct systems design ... this is not accceptable state for production readiness (Context: implement #337 exactly as written.)
Issue #337 "Testing: Scenarios as expected failures (T-03)", as written: Part of the test strategy (captain, 2026-10-05: scenarios that test the changes together; cleaner, faster suites; TESTING.md). Epic #305.
Scope. S2 (counter part), S3 (counter part), S4, S5 ("one pass" part), S6, S8 (drop refusal part), S17 (
views_atpart) written against the target contract and registered asKnownDefect { plan_item: "#315" ... }etc. withReproduction::Deterministic; the parts that hold today (rows, key isolation, diff on replacement) are required passes in the same scenario.Acceptance. Each XFAIL names its issue (V1 #308, V9 #315, V13 #316, V11 #318, V7 #314, V14 #320, V10 #317); the suite prints the XFAIL list at the end; an XPASS fails the job (the mechanism exists in
testkit/src/known_defect.rs); the issue bodies of V9, V11, V13, V14 gain a line "flips scenario Sn to required".The scenarios it names, from the test strategy's catalogue (shop pack: orders, order_item, stock, blocked, link, claim; deployed rules eligible, related (recursive), offer (negation over recursion), n_eligible):
?eligible("o-42", I, W)50 times and?related("i1", X); rule_evaluations delta = 0 and view_reads = reads (V1 counters, V9); rows part required.?stock(I, Q), eligible("o-42", I, _), Q > 5and a session rule joined withoffer; rows equal the recompute answer at the same revision (required); rule_evaluations for eligible = 0, only the query's own operators run (V9).at: delta.revision; at the end a readat: s; query reply revision = at; rows at r_n = snapshot + deltas 1..n; read at s equals the snapshot; a read at a compacted revision failsrevision_compactedand never returns rows (V9, V13); the delta half runs today..subscribeof a join and of a view with a comparison refused withinvalid_requestanddeploy a rule first; the same bodies as?queries work; a valid subscribe afterwards works (V11).offerwith a stricter body in one program with facts gives one atomic old-vs-new delta at its revision, no mixed generations in a concurrent reader (required); droppinglinkwhilerelateddepends on it is refused withconflictnaming the dependents (V7).views_at(V14).What Changed
reads(S2, S3),consistency(S4),fanout(S5),subscribe_rules(S6),generations(S8) andburst(S17). Each runs the part that holds today as a required pass. The rest of the milestone 9 contract is asserted as aReproduction::DeterministicKnownDefectthat names its issue: V1 Baseline benches and counters for the view work (V1) #308, V9 Queries read deployed views from arrangements;revisionon query replies (V9) #315, V13at: <revision>reads with a retention window (V13) #316, V10MaintainedViewstanding queries: subscriptions are diffs of the view between revisions (V10) #317, V11 Keyed subscriptions replace parameterized families; delete the cost probe (V11) #318, V7 Rule generations: build beside, swap at a revision; refuse drops that views depend on (V7) #314 and V14 Write acknowledgement after the maintainer frontier (read-your-writes through views) (V14) #320.KnownDefect::judge_firsthandles checks that hit two issues in turn (counters missing, then the extra work they show).INPUTLAYER_XFAIL_LOG. The crate's own unit tests skip this step.Violation::UnexpectedWork,QueryResult::views_atandWsClient::try_execute_at.supportgains the shared helpersno_rule_evaluations,revision_alignedandNO_VIEW_COUNTERS.make xfail-listtarget.unit-test,e2e-reactiveandci-test-allnow print the open XFAIL list when the run ends, whether it passed or failed, and keep the test exit status.TESTING.mdgets a table of the expected-failure scenarios: for each one, what is required today and which issue each XFAIL waits on.Risk Assessment
✅ Low: The change only touches tests, the testkit and the Makefile/docs. Both fixes from round 1 are correct.
record()skips the file append only when the testkit itself is built under cfg(test), so the scenario binary still logs its XFAILs. S4 now combines its four reads into one result in read order and judges it once, so after V9 alone the read at s still gives an AT_IGNORED XFAIL. Removing the unit-stage log export from ci-test-all leaves e2e-reactive as the only writer, and the final summary prints that list.Testing
I ran the seven new catalogue scenarios in debug against real engine processes, pointing the XFAIL log at the evidence directory. All seven passed their required parts and recorded XFAILs naming #308, #314, #315, #316, #318 and #320. #317 (V10) and #308 are both on S5's counter check, which reports only the first open defect in line, #308; #317 will show once #308 lands.
make xfail-listprinted the list at the end. A temporary edit that made S17's views_at contract hold failed the run with XPASS [#320], and I reverted the edit. The testkit self-tests left the XFAIL log empty, which confirms the review fix. The worktree is clean.Evidence: Scenario run transcript (S2-S6, S8, S17)
Evidence: End-of-run XFAIL list printed by make xfail-list
Evidence: Simulated XPASS fails the run
panicked at testkit/src/known_defect.rs:64:23: XPASS [#320] a write acknowledged during a view rebuild does not carry views_at: the contract now holds. Remove this known-defect marker so the scenario becomes a required pass. test burst::s17_write_burst_reads_its_writes ... FAILEDEvidence: Testkit self-tests do not write to the XFAIL log
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 7 issues found → auto-fixed ✅
Makefile- merge conflict rebasing onto origin/mainTESTING.md- merge conflict rebasing onto origin/maintestkit/src/client.rs- merge conflict rebasing onto origin/maintestkit/src/counters.rs- merge conflict rebasing onto origin/maintestkit/src/fixture.rs- merge conflict rebasing onto origin/maintests/scenarios/harness.rs- merge conflict rebasing onto origin/maintests/scenarios/main.rs- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Review** - 2 issues found → auto-fixed ✅
testkit/src/known_defect.rs:96- The XFAIL list includes fake entries from the testkit's own unit tests.record()appends to theINPUTLAYER_XFAIL_LOGfile whenever that variable is set.make unit-testsets it and then runscargo test --workspace, which includes the testkit self-tests (test_matching_violation_is_expected_failure,test_first_defect_in_front_is_expected_failure,test_defect_behind_shows_once_the_first_is_fixed,test_racy_defect_not_reproduced_passes). So the PR gate (ci.yml:249 runsmake unit-test) prints entries likeXFAIL [X00] test defectandXFAIL [X01] .../NOT REPRODUCED [X00]in 'Expected failures still open'. That breaks the acceptance item 'the suite prints the XFAIL list at the end', because the list is meant to show only real open issues. Fix: skip the file append undercfg!(test)inrecord()(keep the println), or have the self-tests clear the env var. Related site: Makefile:215-217, whereci-test-all's unit stage empties the log and exports the variable but excludes the scenarios binary. The only lines it can collect are this self-test noise, andmake e2e-reactiveempties the log again before the scenarios run. That truncate/export in the unit stage can be dropped. The list is also printed twice inci-test-all: once by e2e-reactive and again at Makefile:300.tests/scenarios/consistency.rs:124- S4 judges eachrevision_alignedoutcome on its own withjudge_first(&[NO_REVISION, AT_IGNORED]). Once V9 Queries read deployed views from arrangements;revisionon query replies (V9) #315 lands but V13at: <revision>reads with a retention window (V13) #316 has not, this fails the job with an XPASS namingat: <revision>reads with a retention window (V13) #316, which is still open. Here is why. Each of the three in-loop readsat: delta.revisionruns right after that delta, with no other writer, so the latest revision is delta.revision. An engine that ignoresatbut now returnsrevisionanswers at exactly that revision with the right rows, so the outcome is Ok.judge_firstsends Ok to the last defect (AT_IGNORED), which panicsXPASS [#316]on the first write, before the readat: s(the one that would really showat: <revision>reads with a retention window (V13) #316) is judged. This contradicts TESTING.md's stated design that 'each stage is an XFAIL of its own issue', and Queries read deployed views from arrangements;revisionon query replies (V9) #315's body, which says it flips only S4'srevisionpart. Fix: fold the four outcomes into one result, keeping the first Err (for examplealigned.into_iter().collect::<Checked<Vec<()>>>().map(drop)), and calljudge_firstonce.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
cargo build --tests --test scenarios -p inputlayerINPUTLAYER_XFAIL_LOG=$E/xfail.log cargo test --test scenarios -- --nocapture reads:: consistency:: fanout:: subscribe_rules:: generations:: burst::(7 passed, each with XFAILs naming their issues)make xfail-list XFAIL_LOG=$E/xfail.log(prints the end-of-run XFAIL list)Adversarial: temporarily replacedNO_VIEWS_AT.judge(views_at)withjudge(Ok(()))in tests/scenarios/burst.rs and rancargo test --test scenarios burst::. The run exited 101 withXPASS [#320]; the edit was reverted with git checkoutINPUTLAYER_XFAIL_LOG=$E/selftest-xfail.log cargo test -p inputlayer-testkit known_defect(self-tests print XFAIL/XPASS but the log file stays 0 bytes)Read-onlygh issue viewfor #315, #316, #318, #320 to confirm the 'flips scenario Sn to required' lines✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.