Repository navigation
test: tier budgets, remaining scenarios, oracle shop corpus and maintained adapter, TESTING.md (T-13..T-16) - #403
Merged
Merged
Conversation
…ained adapter, TESTING.md (T-13..T-16) T-13 (#347): tests/budget.toml and scripts/check-test-budget.py sum each tier's run time from a cargo test log (and the spec runner's new "Specs finished in" line), fail a tier 25 percent over budget and write tests-timing.json; make test-budget runs the PR gate's tests and checks. The perf gate's shop_install fixture installs the shop pack and holds its p50 to an absolute 200 ms ceiling (new [ceilings] in policy.toml). T-14 (#348): scenarios S10/S11 (vectors), S13 (failover), S14 (limits) and S16 (mode matrix, make test-scenarios-modes), each green or an expected failure naming its issue. T-15 (#349): the oracle's corpus gains the shop pack and S9's histories (shared with the scenario), and INPUTLAYER_ORACLE_VIEWS=maintained adds a maintained adapter that reports the mode as unavailable until V2. T-16 (#350): TESTING.md replaced (tiers, decision table, budgets, flake policy, scenario catalogue), CONTRIBUTING pre-PR and testing sections, stale test counts removed from docs/internals/README.md.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the remaining test-strategy issues in one PR: tier budgets (T-13), the remaining scenarios (T-14), the oracle's shop corpus and
maintainedadapter (T-15), and the TESTING.md rewrite (T-16).Closes #347, closes #348, closes #349, closes #350.
CI workflow parts skipped. Hosted CI is paused (#396), so no workflow file changes. The budget step is
make test-budget/scripts/check-test-budget.py; wiring it intoci.ymlafterPer-binary test timesis one step once CI resumes. Checks were run locally (below).T-13 Tier budgets (#347)
tests/budget.toml:unit = 60,component = 180,scenarios = 120,specs = 60,oracle_pr = 20seconds, tolerance 25 percent, and the path-to-tier map (src/unit,tests/scenarios/scenarios,tests/differential_oracle/oracle_pr, othertests/component; doc tests unit). A binary that maps to no tier fails the check.scripts/check-test-budget.pysums each tier from thefinished intime of every binary in acargo testlog and from the spec runner's newSpecs finished inline. It fails a tier over budget + 25 percent, names that tier's slowest binaries, and writestests-timing.json(schemainputlayer.test-timing.v1). It reads downloaded GitHub Actions job logs as-is (timestamps stripped).make test-budgetruns the PR gate's tests (unit-test, then every spec on the debug binaries) and checks them.oracle_pris 20 s, not 15. Firstmate's decision on calibration: CI measured 14.4/16.8/19.9 s before this PR, and the shop corpus adds about 1.3 s.Testslogs (jobs 112041704631, 112043367292, 112065632388) the component tier was 147.4/154.5/184.6 s against its 225 s limit. With 30 s added tows_cancel_testsit reads 177/184/215 s, so the check still passes. Run-to-run noise (147-185 s) rules out tightening the tier instead. The follow-up is a per-binary growth check against a measured baseline: Testing: per-binary test time growth check against a measured baseline (T-13 follow-up) #398. On those same logs (acceptance 1) unit is 56-70 s, scenarios 16-18 s and oracle 14-20 s, all within limits.shop_installis in the perf gate's default fixtures. It installsFixture::shop_pack(Size::Small)(from the testkit, sent asFixture::programs(), the same programsinstallsends) into fresh graphs, 20 per round, checking the anchor rows. A new[ceilings]table inperf-gate/policy.tomlholdsshop_install.install_us.p50to an absolute 200,000 µs. A breach fails the gate whatever the baseline did; a ceiling whose fixture did not run is not judged. The report gets a "Ceilings (absolute)" table, with unit tests for breach, hold and not-run.T-14 Remaining scenarios (#348)
vectors::s10_*nearread evaluates the rule (#327); vector readat: r(#315, then #328)vectors::s11_*.index create view_idx on low_vec(V)is refused "No schema found for relation 'low_vec'"failover::s13_*expect_revisionclaim; after killing the primary the follower holds every acknowledged change, equals both agents' views, refuses writesstore_read_only; a claim at the dead primary's revision is refused and applies nothingPOST /v1/replication/promoteis 404limits::s14_*max_query_memory_bytes(resource_exhausted, in flight during the write), a 1,025-deep term (validation, "nesting exceeds the limit"),top_k3e17 (validation), a 6,000-fact write pastmax_graph_memory_bytes(resource_exhausted, nothing applied); the refused connection keeps answering; all 20 hear the next writemodes::s16_*,make test-scenarios-modesmaintainedmode with identical snapshots, deltas, revisions and write replies (#309: mode not available).make test-scenarios-modesruns the whole suite once per mode,maintainedallowed to fail until V9 (#315)Testkit additions:
EngineBuilder::try_viewsreturningViolation::Unavailable(new variant;views()keeps its panic),Refusal::parse_errors/Refusal::names(a refused unparsable program'svalidation_errors, so S14 can tell the nesting limit from another parse error),Fixture::programspublic.make test-scenariosadded.Findings while writing them (behaviour as it is, asserted as such):
expect_revisionfrom the primary is refusedprecondition_failed("revision 8 has not been issued (latest is 7)") before the read-only check, because the follower counts revisions of its own. S13 accepts either refusal.code(seen while sizing S14's write).view_readsis exported now, so S10's counter check is judged against Keyed top-k and within-radius operator forhnsw_nearestin deployed rules (VV5) #327 directly (the Baseline benches and counters for the view work (V1) #308 stage no longer applies).T-15 Oracle corpus and maintained adapter (#349)
tests/differential_oracle/shop.rsadds the shop pack as a corpus pack: the pack with writes through each construct and a restart, plus S9's and S9b's histories.corpus::check_shop_packruns them through every adapter (shop_pack_corpus_agrees_across_all_adapters), and the reference must model all of it. S9's writes are defined there once:tests/scenarios/retraction.rsnow pairs them with the deltas each step must deliver, andoracle_checktakes the sameHistory.maintainedadapter:RecomputeAdapter::open_maintainedon anEngineHost::open_in(Mode::Maintained).INPUTLAYER_ORACLE_VIEWS=maintained make oracle-testadds it to every history (and runs the soak's server in that mode). On main every oracle test fails withengine.views = "maintained": mode not available on this engine (V2 #309 adds it), never a pass. V2 setsengine.viewsinengine.rs::config.T-16 Documentation (#350)
maketarget it names exists (checked by script). D1's edit at the oldTESTING.md:228("rule catalog changes") is kept.examples/datalog/) is now the tier table.docs/internals/README.md: stale test counts (and the nonexistent01_basics/list) replaced by a pointer to TESTING.md.Local checks (lean debug,
MemoryMax=16G,CARGO_BUILD_JOBS=8)make fmt-checkmake lint(clippy, all targets,-D warnings)make perf-gate-check(perf-gate clippy + 68 unit tests incl. ceiling tests)make pre-pr-selftest(Makefile/scripts changed)make unit-test(whole workspace: 90 binaries incl. scenarios 42 passed / 1 ignored-in-debugsaturation, oracle 10)run_snapshot_tests.sh --debug --skip-build)INPUTLAYER_ORACLE_VIEWS=maintained make oracle-testnesting_depth_tests118 s,wal_transaction_recovery54 s), as TESTING.md says a busy box runs 2-3x the CI runner. The CI logs above are the calibrated measureRUSTDOCFLAGS=-D warnings cargo doc -p inputlayer-testkitmake doc-checkfails on main already (src/storage_engine/snapshot.rs:48private intra-doc link from #390, untouched here)shop_installA/A, release server, 10 rounds, local devbox pinned to 8 cores (load 15-27)One unexplained one-off in the first full run: the oracle's
subscription[group]engine refused a rule insert with ENOENT on corpus case14_aggregations/16_empty_agg_edge_caseswhile a second oracle run was going on. Four oracle reruns and the full rerun passed. Tracked as a flake: #401.Benchmark host: the release
shop_installrun is queued under~/perf-gate-remote/lock(~/bench/t-bundle-b-out/, about 18 jobs ahead); its report goes on this PR as a comment when it lands.