Repository navigation
test: spec corpus hygiene and a generated affected map (T-10) - #393
Merged
Merged
Conversation
…344) Spec corpus (examples/iql), snapshot transcripts 1,153 -> 1,149 (1,152 -> 1,148 specs run by the snapshot runner; the remaining transcript belongs to the 40_load_command helper): - Remove four specs that repeat a lower-numbered spec statement for statement, differing only in the knowledge graph name and, for two of them, the name of the derived relation: 09_recursion/86_descendant = 09_recursion/76_tree_traversal 12_errors/55_empty_batch_insert = 12_errors/41_empty_insert_batch 11_types/34_bool_ops = 11_types/27_bool_logic 22_set_operations/36_exclusive_or = 08_negation/72_exclusive_or - Fold 39_meta_complete into 33_meta: its seven specs move unchanged to 33_meta/33_status .. 39_help_comprehensive. - Move 42_consistency_pack, 43_prompt_integrity and 50_landing_page under 90_product_fixtures; they run as before (the seven landing page demo scripts still have no snapshots and are skipped by the runner). - 50_landing_page/02_incremental.iql no longer claims the second query avoids a full recompute. Affected-only runs: run_snapshot_tests.sh --affected reads the leaf-module map from scripts/affected-map.toml instead of a hand-written case. The map is generated by scripts/gen-affected-map.py from each category's statement types (specs, the files they .load, and recorded transcripts including .debug plan output); --affected refuses a stale map. A change under src/provenance now runs the categories that use .why/.why_not instead of the whole corpus.
… blocked, unresolved
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Do GitHub issue #344 in inputlayer/inputlayer, part of the test restructure in TESTING.md: "Testing: Spec corpus hygiene (T-10)".
Part of the test strategy (captain, 2026-10-05: scenarios that test the changes together; cleaner, faster suites; TESTING.md). Epic #305.
Scope. Remove the 4 exact duplicates (keep the lowest-numbered file), fold
39_meta_completeinto33_meta, move42_consistency_pack,43_prompt_integrity,50_landing_pageunderexamples/iql/90_product_fixtures(still run, named for what they are) and correct50_landing_page/02_incremental.iql's comments per D1; generatescripts/affected-map.tomlfrom each category's statement types (a script that lists whichsrc/modules a category exercises using--debugplan output) sotest-affected.shstops using a hand-writtencase;--debugflag for the runner.Acceptance. 1,153 specs;
test-affected.shpicks the categories for a change insrc/provenance(today: none); runner works against debug binaries.Deliver it as a small PR; done means merged with CI green.
D1 is the docs-truth change (#306, merged as #372): persistent rules are stored definitions re-derived from the base facts on every read, so no doc or example comment may claim materialization or incremental maintenance.
What Changed
examples/iql/: removed four duplicate specs and their snapshots (09_recursion/86_descendant,11_types/34_bool_ops,12_errors/55_empty_batch_insert,22_set_operations/36_exclusive_or), folded39_meta_completeinto33_meta(renumbered33_–39_), and moved42_consistency_pack,43_prompt_integrityand50_landing_pageunder90_product_fixtures/. The comment in50_landing_page/02_incremental.iqlno longer claims the query avoids a full recompute. The differential oracle corpus path, the verified-completions docs and theTESTING.mdcategory table follow the new paths.scripts/gen-affected-map.pywritesscripts/affected-map.toml, andrun_snapshot_tests.sh --affectednow reads leaf-module mappings from it instead of a hand-writtencase.src/provenancenow maps to36_explain_traceand90_product_fixtures, and the vector and temporal modules pick up more categories than the old hand-written list. Unmappedsrc/changes still run the whole corpus, as do changes to the map or its generator..loaded files and recorded transcripts against a fixed statement-type → module table, rather than using--debugplan output; its docstring records why (plans name nosrc/module, and.why/.why_notproduce no plan). No--debugflag is added to the runner or the generator in this branch. The map and generator are added to the CI rust path filter and to themake pre-prrust inputs.🤖 Generated with Claude Code
Risk Assessment
Testing
Built the debug server and client through the runner (
test-affected.sh HEAD --debug) and drove everything on an isolated port and data directory. A probe edit insrc/provenanceselected36_explain_traceand90_product_fixturesand those specs passed ontarget/debug. Further probe edits covered docs-only, vector, temporal, mixed, prefix-collision and map-file changes; each selected the expected categories or widened to the whole corpus. The folded33_metaspecs, the moved fixtures and the four kept twins passed one by one. The whole corpus on debug binaries failed once on a 60 s client timeout for the consistency-pack fixture while host load was above 40 on 32 cores; the identical re-run passed with 1,148 passed and 7 skipped, and the fixture alone takes about 8 s. The seven landing-page scripts are skipped by the runner because they have no recorded transcript, as at the base, so I ran02_incremental.iqldirectly against a disposable debug server. The oracle corpus test,example_verification, generator regeneration andmake pre-prrouting also passed. No UI surface is involved, so there are no screenshots. All probe edits were reverted and build output removed; the worktree is clean.python3 scripts/gen-affected-map.pyleft no git diff;--debugprinted 'Unknown option: --debug', exit 1Evidence: Provenance edit selects its categories and runs them on debug binaries
$ git status --short M src/provenance/why_not.rs $ INPUTLAYER_TEST_PORT=18744 scripts/test-affected.sh HEAD --debug Changes since HEAD affect these categories: - 36_explain_trace - 90_product_fixtures Building project... Starting server... Server started (PID 1746769) Running 25 tests in parallel (32 workers), 0 sequentially... Passed: 18 Failed: 0 Skipped: 7 All tests passed! exit=0Evidence: Corpus shape: counts, removed duplicates, folded and moved categories
Evidence: Affected-selection cases, including adversarial prefix and widening cases
Evidence: Folded 33_meta specs, moved fixtures and kept twins, one by one on debug binaries
PASS [08_negation] 72_exclusive_or.iql PASS [09_recursion] 76_tree_traversal.iql PASS [11_types] 27_bool_logic.iql PASS [12_errors] 41_empty_insert_batch.iql PASS [33_meta] 33_status.iql PASS [33_meta] 34_load_basic.iql PASS [33_meta] 35_rule_management.iql PASS [33_meta] 36_rel_operations.iql PASS [33_meta] 37_session_complete.iql PASS [33_meta] 38_kg_comprehensive.iql PASS [33_meta] 39_help_comprehensive.iql PASS [42_consistency_pack] 01_consistency_core_fixture.iql PASS [43_prompt_integrity] 01_prompt_integrity_fixture.iql SKIP [50_landing_page] 01_retraction.iql - no .out file (6 more landing-page SKIP lines) Passed: 13 Failed: 0 Skipped: 7Evidence: Full corpus on debug binaries, first run (one timeout under host load)
Evidence: Full corpus on debug binaries, re-run (all pass)
$ scripts/run_snapshot_tests.sh --debug --skip-build (second run) Running 1151 tests in parallel (32 workers), 4 sequentially... Running 4 sequential tests (global state)... PASS [01_knowledge_graph] 02_list_kg.iql PASS [33_meta] 03_compact.iql PASS [33_meta] 12_compact_empty.iql PASS [33_meta] 38_kg_comprehensive.iql Passed: 1148 Failed: 0 Skipped: 7 All tests passed! exit=0Evidence: Oracle corpus test and example verification
Evidence: make pre-pr routing for the map files
scripts/affected-map.toml: exit=0 checks=['clippy', 'fmt', 'perf-gate.sh', 'python3', 'test', 'test-affected.sh'] scripts/gen-affected-map.py: exit=0 checks=['clippy', 'fmt', 'perf-gate.sh', 'python3', 'test', 'test-affected.sh'] scripts/change.sh: exit=0 checks=['fmt', 'perf-gate.sh', 'python3']Evidence: Landing-page 02_incremental.iql run live against a debug server
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
examples/iql/09_recursion/86_descendant.iql:1- Intent conformance: the acceptance criterion says "1,153 specs", but the change leaves 1,149 recorded transcripts (1,148 runnable by the snapshot runner). The base commit already has exactly 1,153.iql.outfiles (1,160.iql), so the other required criterion, "Remove the 4 exact duplicates", necessarily takes it to 1,149; the commit message states "1,153 -> 1,149" itself. The two criteria cannot both hold at this base (the issue's number looks like it was written against a 1,157-spec corpus). Needs a human decision: accept 1,149 as the intended end state, or identify which removals should not happen.scripts/gen-affected-map.py:35- Intent conformance: the intent asks for "a script that lists whichsrc/modules a category exercises using--debugplan output". The generator does not run anything with--debugor derive modules from plans:STATEMENT_TYPES(lines 35-42) is a hand-written table of three regexes over spec/transcript text, each mapped to a hard-coded module list, so the statement-type to module mapping is still hand-maintained (moved from the bashcaseinto Python). Recorded.debugtranscripts are only incidentally included in the text the regexes scan. The--debugoption the change adds (lines 15, 77-78, 97, 100) belongs to the generator and only prints each category's matched statement types; the runner's--debugflag already existed at the base (from ci: PR gate runs scenarios, affected specs and the oracle from one debug build; add main.yml on push to main #370) and is untouched. The category sets it produces are sound for the three types (I checked the regexes against the builtin function table and the.why/.why_not/.why fullparser, andsrc/provenanceis only reached from the handler's why/why_not paths), so this is a question of whether the regex-over-text approach satisfies the criterion, not a wrong-result defect.scripts/run_snapshot_tests.sh:527- Simplification: the stale-map refusal (gen-affected-map.py --checkbefore every--affectedrun, lines 527-530) is a component no intent requirement asks for. It is a new hard gate: CI runsrun_snapshot_tests.sh --debug --skip-build --affected HEAD^1(.github/workflows/ci.yml:253), so any PR that adds or changes a spec so that a category gains or loses a.why/vector/temporal statement now fails CI until the map is regenerated and committed, and every--affectedrun now needs python3. The intent only requires that the map be generated and thattest-affected.shread it. Remedy if not wanted: remove the check (and the--checkmode at gen-affected-map.py:101-107); otherwise confirm the gate is intended.🔧 Fix applied.
3 warnings still open:
examples/iql/09_recursion/86_descendant.iql:1- Intent conformance: the acceptance criterion says "1,153 specs", but the change leaves 1,149 recorded transcripts (1,148 runnable by the snapshot runner). The base commit already has exactly 1,153.iql.outfiles (1,160.iql), so the other required criterion, "Remove the 4 exact duplicates", necessarily takes it to 1,149; the commit message states "1,153 -> 1,149" itself. The two criteria cannot both hold at this base (the issue's number looks like it was written against a 1,157-spec corpus). Needs a human decision: accept 1,149 as the intended end state, or identify which removals should not happen.scripts/gen-affected-map.py:35- Intent conformance: the intent asks for "a script that lists whichsrc/modules a category exercises using--debugplan output". The generator does not run anything with--debugor derive modules from plans:STATEMENT_TYPES(lines 35-42) is a hand-written table of three regexes over spec/transcript text, each mapped to a hard-coded module list, so the statement-type to module mapping is still hand-maintained (moved from the bashcaseinto Python). Recorded.debugtranscripts are only incidentally included in the text the regexes scan. The--debugoption the change adds (lines 15, 77-78, 97, 100) belongs to the generator and only prints each category's matched statement types; the runner's--debugflag already existed at the base (from ci: PR gate runs scenarios, affected specs and the oracle from one debug build; add main.yml on push to main #370) and is untouched. The category sets it produces are sound for the three types (I checked the regexes against the builtin function table and the.why/.why_not/.why fullparser, andsrc/provenanceis only reached from the handler's why/why_not paths), so this is a question of whether the regex-over-text approach satisfies the criterion, not a wrong-result defect.scripts/gen-affected-map.py:34- Recorded human decision not met (round 1,map-not-from-debug-plan: "Have the generator derive each category's src/ modules from the recorded --debug plan output of its specs rather than from regexes over spec text ... If the --debug plan output cannot support this at all, stop and report what it does expose instead of approximating"). The round 1 fix commit (20a42fe, titled "plan-derived affected map blocked, unresolved") only removed the stale-map gate;STATEMENT_TYPES(lines 34-41) is unchanged: three hand-written regexes over spec/transcript text, each mapped to a hard-coded module list, andscripts/affected-map.tomlis still produced from them. The fixer stopped as the decision instructed, so this is the report of what the plan output exposes, verified in the source: (1) recorded plan output exists for only 7 specs, all in33_meta(16_debug,17_debug_join,18_debug_recursive,19_debug_with_filter,20_debug_negation,21_debug_aggregation,26_debug_complex); the other 35 categories record none. (2) That output is aPIPELINE TRACEof parsed rules and IR operators (Scan,Join,Map,Union,Filter,Antijoin,Aggregate); it names nosrc/module. (3).why/.why_notare meta commands that never produce a plan, sosrc/provenancecannot be derived from plan output at all; the acceptance "a change insrc/provenanceselects the categories that exercise it" currently holds only through thewhyregex. (4) The runner's--debugflag (run_snapshot_tests.sh:488) selectstarget/debugbinaries and emits no plan output. Meeting the decision as written would need new machinery (the generator running every spec's queries through.debugagainst a live server, plus a separate non-plan source for provenance), which extends the change; that remedy, not the defect, needs authorization. Decision needed: accept the regex-over-text generator as the end state, or authorize the larger plan-running generator. Also inside this component and not required by the intent: the generator's own--debugoption (lines 14, 76-77, 95, 98), which only prints each category's matched types.🔧 Fix applied.
3 issues (2 warnings, 1 info) still open:
examples/iql/09_recursion/86_descendant.iql:1- Intent conformance: the acceptance criterion says "1,153 specs", but the change leaves 1,149 recorded transcripts (1,148 runnable by the snapshot runner). The base commit already has exactly 1,153.iql.outfiles (1,160.iql), so the other required criterion, "Remove the 4 exact duplicates", necessarily takes it to 1,149; the commit message states "1,153 -> 1,149" itself. The two criteria cannot both hold at this base (the issue's number looks like it was written against a 1,157-spec corpus). Needs a human decision: accept 1,149 as the intended end state, or identify which removals should not happen.scripts/run_snapshot_tests.sh:469- The change declaresscripts/affected-map.tomlandscripts/gen-affected-map.pyas inputs that affect every spec in two places (run_snapshot_tests.sh:469 sets AFFECTED_ALL=1 for them; Makefile:644 adds them to PRE_PR_RUST_INPUTS), but the third owner of the same input list, therustpaths filter in .github/workflows/ci.yml:45-64, still lists onlyscripts/run_snapshot_tests.shandscripts/test-timings.sh. Concrete sequence: a PR edits only the generator's STATEMENT_TYPES table and commits the regenerated map (for example dropping a category fromsrc/provenance); no path in therustfilter matches, so the whole Rust job, includingrun_snapshot_tests.sh --debug --skip-build --affected HEAD^1at ci.yml:253, is skipped and the changed selection is never exercised in CI, although the runner itself would have run the full corpus. Fix: addscripts/affected-map.tomlandscripts/gen-affected-map.pyto therustfilter in ci.yml next toscripts/run_snapshot_tests.sh.examples/iql/33_meta/34_load_basic.iql:1- The folded spec's header still reads "Basic operations in meta_complete category", a category this change removes. It is a comment only (comments are not in the recorded transcript, so the snapshot is unaffected); the files were moved unchanged on purpose, so this is noted rather than prescribed.🔧 Fix applied.
3 issues (1 warning, 2 infos) still open:
examples/iql/09_recursion/86_descendant.iql:1- Intent conformance: the acceptance criterion says "1,153 specs", but the change leaves 1,149 recorded transcripts (1,148 runnable by the snapshot runner). The base commit already has exactly 1,153.iql.outfiles (1,160.iql), so the other required criterion, "Remove the 4 exact duplicates", necessarily takes it to 1,149; the commit message states "1,153 -> 1,149" itself. The two criteria cannot both hold at this base (the issue's number looks like it was written against a 1,157-spec corpus). Needs a human decision: accept 1,149 as the intended end state, or identify which removals should not happen.examples/iql/33_meta/34_load_basic.iql:1- The folded spec's header still reads "Basic operations in meta_complete category", a category this change removes. It is a comment only (comments are not in the recorded transcript, so the snapshot is unaffected); the files were moved unchanged on purpose, so this is noted rather than prescribed.examples/iql/09_recursion/86_descendant.iql:1- For awareness only: the four removed specs are not byte-identical to the specs kept, and no byte-identical spec pair exists at the base. Each repeats a lower-numbered spec statement for statement and records the same results, differing only in comments, the knowledge graph name and, for two, the derived relation name:09_recursion/86_descendant=76_tree_traversal;12_errors/55_empty_batch_insert=41_empty_insert_batch;11_types/34_bool_ops=27_bool_logic(relationstrue_flags/false_flagsvsenabled/disabled);22_set_operations/36_exclusive_or=08_negation/72_exclusive_or(xor_abvsxor_result, kept twin is in another category). I diffed all four pairs at the base, sources and transcripts; the lowest-numbered file is kept in each. This is the only reading of "the 4 exact duplicates" the corpus supports, and the recorded decision already accepts the 1,149 end state.examples/iql/90_product_fixtures/42_consistency_pack/01_consistency_core_fixture.iql:1- In the first full-corpus run on debug binaries (32 parallel jobs, host load average above 40 on 32 cores from other work on the machine), this spec hit the runner's 60 s client timeout just before its final.kg dropline and was reported as FAIL. The identical re-run passed (1,148 passed, 0 failed), and the spec alone finishes in about 8 s. The engine sources and the spec are unchanged by this change (the file was only moved), so I read this as a load-dependent flake rather than a regression; the move does place it at the end of the parallel schedule instead of the middle. Evidence: 05-full-corpus-debug.txt and 06-full-corpus-debug-rerun.txt.python3 scripts/gen-affected-map.pyleft no git diff;--debugprinted 'Unknown option: --debug', exit 1INPUTLAYER_TEST_PORT=18744 scripts/test-affected.sh HEAD --debugwith a probe edit insrc/provenance/why_not.rs(builds debug binaries, runs the selected categories)scripts/test-affected.sh HEAD --debug --skip-buildwith a docs-only edit (selects nothing, exit 0)scripts/test-affected.sh HEAD --skip-buildwith probe edits: provenance plus a29_temporalspec, new filesrc/provenance_probe.rs, provenance plussrc/lib.rs,scripts/affected-map.toml,src/vector_ops.rs(selection printed; stops at missing release binaries)scripts/test-affected.sh HEAD --debug --skip-buildwith a probe edit insrc/temporal_ops.rs(125 specs pass)scripts/run_snapshot_tests.sh --debug --skip-build -j 1 -f '/\(33_meta/3[3-9]_\|90_product_fixtures\|76_tree_traversal\|41_empty_insert_batch\|27_bool_logic\|08_negation/72_exclusive_or\)'scripts/run_snapshot_tests.sh --debug --skip-buildtwice (first run: one timeout; second run: 1,148 passed, 7 skipped)scripts/run_snapshot_tests.sh --debug --skip-build -j 1 -f 42_consistency_pack(passes in about 8 s)python3 scripts/gen-affected-map.pythengit diff --stat -- scripts/affected-map.toml(no difference);python3 scripts/gen-affected-map.py --debug(rejected, exit 1)cargo test --test differential_oracle iql_corpus_agrees_with_spec_and_referencecargo test --test example_verificationmake pre-prin a disposable repo with stubbed tools viascripts/test_pre_pr.py'srun_gate, for changes toscripts/affected-map.toml,scripts/gen-affected-map.pyand a control scripttarget/debug/inputlayer-client --script examples/iql/90_product_fixtures/50_landing_page/02_incremental.iqlagainst a disposable debug server on port 18745Corpus shape: counts of.iql.outat base and HEAD, presence of removed duplicates and kept twins, old category directories goneTESTING.md:476- The category table in TESTING.md has no rows for three categories that exist in examples/iql: 34_type_conversion, 36_explain_trace and 41_timing_breakdown. They were already missing at the base commit, so I left them alone; the change's own rows (33_meta, 90_product_fixtures, removal of 39_meta_complete) are correct. Possible follow-up: add the three rows, or generate the table from the directory listing.tests/differential_oracle/corpus.rs:36-make lint(cargo clippy over the whole workspace) was not run in this pass. The only Rust change is one string literal in this file. Checks that did run and pass:cargo fmt --all -- --check,bash -n scripts/run_snapshot_tests.sh,python3 -m py_compile scripts/gen-affected-map.py, and scripts/affected-map.toml matches the generator's output. ruff and shellcheck are not on PATH and the repository configures neither.✅ **Push** - passed
✅ No issues found.