Skip to content

ci: gate the .debug query-plan snapshot specs on every PR - #362

Closed
jsam wants to merge 1 commit into
mainfrom
fm/fix-snapshot-specs
Closed

ci: gate the .debug query-plan snapshot specs on every PR#362
jsam wants to merge 1 commit into
mainfrom
fm/fix-snapshot-specs

Conversation

@jsam

@jsam jsam commented Oct 5, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Intent

I want you to really hyper focus on making the symbolic engine, you know, crisp and really truly production ready. ... just ensure that both machines are propelry utilized now (Context: implement #290 exactly as written.)

Issue #290, "Snapshot specs 33_meta/17, 18, 26 (.debug plans) fail on main since #254's join-planner change", as written: Found by the EN-15 lane while rebasing: three query-plan snapshot specs (33_meta/17, 33_meta/18, 33_meta/26, .debug plan output) fail on main since the join-planner change in #254. CI's full suite is green, so these specs are not in the CI gate. Fix: confirm the new plans are correct, update the snapshots (or fix the planner if a plan regressed), and add these specs to CI so plan output is gated.

What Changed

  • Added a make plan-snapshots target. It runs the .debug query-plan specs 33_meta/17_debug_join, 18_debug_recursive and 26_debug_complex sequentially, using the dev binaries that are already built. The fast-gate test job in .github/workflows/ci.yml now runs it right after make unit-test. The workflow's Rust path filter also now includes scripts/run_snapshot_tests.sh.
  • Changed scripts/run_snapshot_tests.sh:
    • New --profile NAME option, default release. dev resolves to target/debug.
    • When a -f filter matches no specs, the script now fails instead of passing with nothing run, so a renamed spec can't silently leave the gate.
  • Documented the new target, the --skip-build and --profile options, and the extra fast-gate step in TESTING.md and CONTRIBUTING.

Risk Assessment

✅ Low: This change only touches CI and tests. The new --profile option keeps release as its default and maps dev to target/debug. make plan-snapshots expands to a filter that matches exactly the three .debug specs, and those specs' .iql.out files were already regenerated on main by #282. The fast-gate job builds the dev binaries with cargo test, which the step needs, and the empty-filter guard works because the script does not run under set -e.

Testing

I built the dev server and client binaries and drove make plan-snapshots against the real inputlayer server. On the current snapshots, specs 17, 18 and 26 all pass. To check the gate catches drift, I changed one line in the 26 snapshot; that spec then failed and make exited non-zero. A filter matching no specs also fails instead of passing with nothing run. Running the target on the binaries built by the same build step make unit-test uses also passes. I parsed the CI workflow and confirmed the new step runs after the unit tests on PRs to main; that step has not run live on GitHub Actions, so the scenario is marked untested. All seven 33_meta debug specs pass too. I removed the build output afterwards, and the worktree is clean.

  • Live validation: ✅ go - 4 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Developer runs make plan-snapshots: specs 33_meta/17, 18 and 26 run against the real server and all pass ✅ pass live plan-snapshots-pass.log: PASS 17_debug_join, 18_debug_recursive, 26_debug_complex; 3 passed, exit 0
Adversarial: plan output no longer matches a snapshot, so make plan-snapshots fails ✅ pass live plan-snapshots-drift-detected.log: FAIL 26_debug_complex with a diff on the Join line, make Error 1
Adversarial: a filter that matches no specs (such as a renamed spec) fails instead of passing ✅ pass live empty-filter.log: 'No tests match filter', exit 1
CI order: binaries built by cargo test --workspace --all-features, then make plan-snapshots --skip-build reuses them and passes ✅ pass live ci-sequence-plan-snapshots.log: 3 passed after deleting the debug binaries and rebuilding them with cargo test --no-run
The PR workflow runs the plan-snapshots step after the unit tests in the test job, and triggers on changes to the snapshot script ⏸️ untested no The prior payload did not establish a live result. This scenario can only run live on GitHub Actions, which a PR triggers, and the CI phase owns that. The only check was a static parse of the workflow…
Evidence: make plan-snapshots passing on the 3 plan specs 
./scripts/run_snapshot_tests.sh --skip-build --profile dev -j 1 -f '33_meta/\(17_debug_join\|18_debug_recursive\|26_debug_complex\)\.iql$'
Skipping build (--skip-build)...
Starting server...
Server log: /tmp/tmp.FTuf43t2sE/server.log
Server started (PID 1932278)
Pre-cleaning stale knowledge graphs...

========================================
  InputLayer Snapshot Tests (sequential)
========================================

PASS [33_meta] 17_debug_join.iql
PASS [33_meta] 18_debug_recursive.iql
PASS [33_meta] 26_debug_complex.iql

========================================
  Summary
========================================
Passed:  3
Failed:  0
Skipped: 0

All tests passed!
Evidence: Gate fails when a plan snapshot differs (26_debug_complex) 
./scripts/run_snapshot_tests.sh --skip-build --profile dev -j 1 -f '33_meta/\(17_debug_join\|18_debug_recursive\|26_debug_complex\)\.iql$'
Skipping build (--skip-build)...
Starting server...
Server log: /tmp/tmp.3KFwbsgyFE/server.log
Server started (PID 1941426)
Pre-cleaning stale knowledge graphs...

========================================
  InputLayer Snapshot Tests (sequential)
========================================

PASS [33_meta] 17_debug_join.iql
PASS [33_meta] 18_debug_recursive.iql
FAIL [33_meta] 26_debug_complex.iql
  --- Diff (expected vs actual) ---
33c33
<       JOINX[L:[1], R:[0]] -> [SId, PId, Price, _ph_a1_c1]
---
>       Join[L:[1], R:[0]] -> [SId, PId, Price, _ph_a1_c1]


========================================
  Summary
========================================
Passed:  2
Failed:  1
Skipped: 0

Some tests failed!
Run with -v for detailed output

=== Server Log (last 50 lines) ===
�[2m2026-10-05T18:11:50.551423Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m snapshot_publish_complete �[3mrelations�[0m�[2m=�[0m2 �[3mrules�[0m�[2m=�[0m0 �[3msnapshot_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.551502Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_statement_exec_complete �[3mprogram_len�[0m�[2m=�[0m33 �[3mstmt_exec_ms�[0m�[2m=�[0m4 �[3msession_facts�[0m�[2m=�[0m0 �[3msession_rules�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.551531Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_complete_messages �[3mprogram_len�[0m�[2m=�[0m33 �[3mtotal_ms�[0m�[2m=�[0m4
�[2m2026-10-05T18:11:50.551977Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_complete �[3mprogram_len�[0m�[2m=�[0m33 �[3mcompute_ms�[0m�[2m=�[0m5 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m4 �[3mmsg_bytes�[0m�[2m=�[0m73�[0m
�[2m2026-10-05T18:11:50.552082Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_end �[3mprogram_len�[0m�[2m=�[0m33 �[3melapsed_ms�[0m�[2m=�[0m5 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m4 �[3mmsg_bytes�[0m�[2m=�[0m73�[0m
�[2m2026-10-05T18:11:50.553551Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_start �[3mprogram_len�[0m�[2m=�[0m35 �[3mparam_count�[0m�[2m=�[0m0 �[3mprogram_preview�[0m�[2m=�[0m+order_item[(101, 201), (102, 201)] �[2m�[3mrequest_id�[0m�[2m=�[0m5 �[3mmsg_bytes�[0m�[2m=�[0m75�[0m
�[2m2026-10-05T18:11:50.553907Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_start �[3mprogram_len�[0m�[2m=�[0m35
�[2m2026-10-05T18:11:50.554012Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mtest_explain_complex_n33t26 �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.554050Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_parse_complete �[3mprogram_len�[0m�[2m=�[0m35 �[3mstatements�[0m�[2m=�[0m1 �[3mparse_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.559005Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine::program_commit�[0m�[2m:�[0m program_commit_persisted �[3mkg�[0m�[2m=�[0mtest_explain_complex_n33t26 �[3mrelations�[0m�[2m=�[0m1 �[3mstatements�[0m�[2m=�[0m1 �[3mtime�[0m�[2m=�[0m10 �[3mpersist_ms�[0m�[2m=�[0m4
�[2m2026-10-05T18:11:50.559269Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m snapshot_publish_complete �[3mrelations�[0m�[2m=�[0m3 �[3mrules�[0m�[2m=�[0m0 �[3msnapshot_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.559381Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_statement_exec_complete �[3mprogram_len�[0m�[2m=�[0m35 �[3mstmt_exec_ms�[0m�[2m=�[0m5 �[3msession_facts�[0m�[2m=�[0m0 �[3msession_rules�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.559426Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_complete_messages �[3mprogram_len�[0m�[2m=�[0m35 �[3mtotal_ms�[0m�[2m=�[0m5
�[2m2026-10-05T18:11:50.560124Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_complete �[3mprogram_len�[0m�[2m=�[0m35 �[3mcompute_ms�[0m�[2m=�[0m6 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m5 �[3mmsg_bytes�[0m�[2m=�[0m75�[0m
�[2m2026-10-05T18:11:50.560224Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_end �[3mprogram_len�[0m�[2m=�[0m35 �[3melapsed_ms�[0m�[2m=�[0m6 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m5 �[3mmsg_bytes�[0m�[2m=�[0m75�[0m
�[2m2026-10-05T18:11:50.564068Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_start �[3mprogram_len�[0m�[2m=�[0m71 �[3mparam_count�[0m�[2m=�[0m0 �[3mprogram_preview�[0m�[2m=�[0m.debug ?product(PId, Name), sale(SId, PId, Price), order_item(SId, CId) �[2m�[3mrequest_id�[0m�[2m=�[0m6 �[3mmsg_bytes�[0m�[2m=�[0m111�[0m
�[2m2026-10-05T18:11:50.565543Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_start �[3mprogram_len�[0m�[2m=�[0m71
�[2m2026-10-05T18:11:50.565593Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mtest_explain_complex_n33t26 �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.565620Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_parse_complete �[3mprogram_len�[0m�[2m=�[0m71 �[3mstatements�[0m�[2m=�[0m1 �[3mparse_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.565896Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mtest_explain_complex_n33t26 �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.569226Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_statement_exec_complete �[3mprogram_len�[0m�[2m=�[0m71 �[3mstmt_exec_ms�[0m�[2m=�[0m3 �[3msession_facts�[0m�[2m=�[0m0 �[3msession_rules�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.569278Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_complete_messages �[3mprogram_len�[0m�[2m=�[0m71 �[3mtotal_ms�[0m�[2m=�[0m3
�[2m2026-10-05T18:11:50.569642Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_complete �[3mprogram_len�[0m�[2m=�[0m71 �[3mcompute_ms�[0m�[2m=�[0m4 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m6 �[3mmsg_bytes�[0m�[2m=�[0m111�[0m
�[2m2026-10-05T18:11:50.569737Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_end �[3mprogram_len�[0m�[2m=�[0m71 �[3melapsed_ms�[0m�[2m=�[0m5 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m6 �[3mmsg_bytes�[0m�[2m=�[0m111�[0m
�[2m2026-10-05T18:11:50.571485Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_start �[3mprogram_len�[0m�[2m=�[0m15 �[3mparam_count�[0m�[2m=�[0m0 �[3mprogram_preview�[0m�[2m=�[0m.kg use default �[2m�[3mrequest_id�[0m�[2m=�[0m7 �[3mmsg_bytes�[0m�[2m=�[0m55�[0m
�[2m2026-10-05T18:11:50.571755Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_start �[3mprogram_len�[0m�[2m=�[0m15
�[2m2026-10-05T18:11:50.571834Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mtest_explain_complex_n33t26 �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.571897Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_parse_complete �[3mprogram_len�[0m�[2m=�[0m15 �[3mstatements�[0m�[2m=�[0m1 �[3mparse_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.571928Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m meta_kg_use_start �[3mkg�[0m�[2m=�[0mdefault
�[2m2026-10-05T18:11:50.571945Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mdefault �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.571967Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m meta_kg_use_ok �[3mkg�[0m�[2m=�[0mdefault
�[2m2026-10-05T18:11:50.571988Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_complete_messages �[3mprogram_len�[0m�[2m=�[0m15 �[3mtotal_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.572626Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_complete �[3mprogram_len�[0m�[2m=�[0m15 �[3mcompute_ms�[0m�[2m=�[0m0 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m7 �[3mmsg_bytes�[0m�[2m=�[0m55�[0m
�[2m2026-10-05T18:11:50.572688Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_end �[3mprogram_len�[0m�[2m=�[0m15 �[3melapsed_ms�[0m�[2m=�[0m1 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m7 �[3mmsg_bytes�[0m�[2m=�[0m55�[0m
�[2m2026-10-05T18:11:50.574104Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_start �[3mprogram_len�[0m�[2m=�[0m36 �[3mparam_count�[0m�[2m=�[0m0 �[3mprogram_preview�[0m�[2m=�[0m.kg drop test_explain_complex_n33t26 �[2m�[3mrequest_id�[0m�[2m=�[0m8 �[3mmsg_bytes�[0m�[2m=�[0m76�[0m
�[2m2026-10-05T18:11:50.574600Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_start �[3mprogram_len�[0m�[2m=�[0m36
�[2m2026-10-05T18:11:50.574651Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mdefault �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.574681Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_parse_complete �[3mprogram_len�[0m�[2m=�[0m36 �[3mstatements�[0m�[2m=�[0m1 �[3mparse_ms�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.574708Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m meta_kg_drop_start �[3mkg�[0m�[2m=�[0mtest_explain_complex_n33t26
�[2m2026-10-05T18:11:50.585318Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m save_metadata_complete �[3melapsed_ms�[0m�[2m=�[0m3
�[2m2026-10-05T18:11:50.586091Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_drop_prepare_complete �[3mkg�[0m�[2m=�[0mtest_explain_complex_n33t26 �[3melapsed_ms�[0m�[2m=�[0m11
�[2m2026-10-05T18:11:50.586114Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m meta_kg_drop_prepare_ok �[3mkg�[0m�[2m=�[0mtest_explain_complex_n33t26
�[2m2026-10-05T18:11:50.606468Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m save_metadata_complete �[3melapsed_ms�[0m�[2m=�[0m4
�[2m2026-10-05T18:11:50.610304Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_drop_finish_complete �[3mkg�[0m�[2m=�[0mtest_explain_complex_n33t26 �[3melapsed_ms�[0m�[2m=�[0m24
�[2m2026-10-05T18:11:50.610369Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m meta_kg_drop_finish_ok �[3mkg�[0m�[2m=�[0mtest_explain_complex_n33t26
�[2m2026-10-05T18:11:50.610457Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_statement_exec_complete �[3mprogram_len�[0m�[2m=�[0m36 �[3mstmt_exec_ms�[0m�[2m=�[0m35 �[3msession_facts�[0m�[2m=�[0m0 �[3msession_rules�[0m�[2m=�[0m0
�[2m2026-10-05T18:11:50.610479Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_complete_messages �[3mprogram_len�[0m�[2m=�[0m36 �[3mtotal_ms�[0m�[2m=�[0m36
�[2m2026-10-05T18:11:50.610918Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_complete �[3mprogram_len�[0m�[2m=�[0m36 �[3mcompute_ms�[0m�[2m=�[0m36 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m8 �[3mmsg_bytes�[0m�[2m=�[0m76�[0m
�[2m2026-10-05T18:11:50.611247Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_end �[3mprogram_len�[0m�[2m=�[0m36 �[3melapsed_ms�[0m�[2m=�[0m37 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m8 �[3mmsg_bytes�[0m�[2m=�[0m76�[0m
�[2m2026-10-05T18:11:50.614323Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::rest::handlers::ws�[0m�[2m:�[0m ws_session_disconnecting �[3mactive_sessions�[0m�[2m=�[0m1
=== End Server Log ===
make: *** [Makefile:325: plan-snapshots] Error 1
Evidence: Filter matching no specs fails 
Skipping build (--skip-build)...
Starting server...
Server log: /tmp/tmp.0mKyp9Cvap/server.log
Server started (PID 1947521)
Pre-cleaning stale knowledge graphs...

========================================
  InputLayer Snapshot Tests (sequential)
========================================

No tests match filter: 33_meta/99_nope\.iql$
Evidence: make plan-snapshots after the cargo test --all-features build (CI order) 
PASS [33_meta] 17_debug_join.iql
PASS [33_meta] 18_debug_recursive.iql
PASS [33_meta] 26_debug_complex.iql
Passed:  3
Failed:  0
All tests passed!
Evidence: Parsed CI workflow: plan-snapshots step in the test job, path filter 
job: test if: needs.changes.outputs.rust == 'true' && !contains(github.event.pull_request.labels.*.name, 'release') && !(startsWith(github.head_ref, 'release/') &&
  github.event.pull_request.head.repo.full_name == github.repository) needs: changes
  step: None | run: None
  step: Install Rust toolchain | run: None
  step: Cache cargo | run: None
  step: Run unit + integration tests | run: make unit-test
  step: Run query-plan snapshots | run: make plan-snapshots
filter job changes : ['rust:', "  - 'scripts/run_snapshot_tests.sh'", 'python:', 'js:']
on: {'pull_request': {'branches': ['main']}}

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found ✅
  • ℹ️ Makefile:323 - The new empty-filter guard (scripts/run_snapshot_tests.sh:558-563) says it stops 'a renamed spec drops out of its gate', but it only fires when all three plan specs vanish. If only one of 17_debug_join / 18_debug_recursive / 26_debug_complex is renamed, PLAN_SNAPSHOTS still matches the other two. TEST_TOTAL is then 2, and the gate passes while silently no longer checking the renamed plan. Closing this would mean asserting an expected spec count, for example a new runner option or a count check in the Makefile target. That adds to the change, so it is noted rather than prescribed.

✅ No issues found.

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 5 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Developer runs make plan-snapshots after building dev binaries: only the 3 .debug plan specs run, and all pass ✅ pass live 01-plan-snapshots-pass.txt: PASS 17_debug_join, 18_debug_recursive, 26_debug_complex; Passed 3, Failed 0; exit 0
A planner change that alters a .debug plan fails the gate (the snapshot is changed so it no longer matches the engine's output) ✅ pass live 02-perturbed-plan-fails.txt: FAIL [33_meta] 26_debug_complex.iql with the changed Join line in the diff; make exit 2
A renamed or missing spec cannot pass vacuously: a filter matching no spec exits non-zero ✅ pass live 03-empty-filter-fails.txt: 'No tests match filter', exit 1
--profile selects the binary directory; a profile with no built binaries fails clearly ✅ pass live 04-missing-profile-fails.txt: 'Binaries not found' under target/nosuch, exit 1; dev mapped to target/debug in scenario 1
The fast-gate PR CI test job runs make plan-snapshots after make unit-test ⏸️ untested no This only runs on GitHub Actions for a PR to main, which the outer CI phase owns. I parsed the workflow's structure but did not run it live.
The regenerated plans are correct: 26_debug_complex no longer builds a cross-product join ✅ pass live The live engine's .debug output matches the committed snapshot (scenario 1), which contains no Join[L:[], R:[]]
  • cargo build --workspace --all-features --bin inputlayer-client --bin inputlayer-server (dev profile, the binaries make unit-test leaves behind)
  • make plan-snapshots (passes 3/3: 17_debug_join, 18_debug_recursive, 26_debug_complex; 17_clear_prefix correctly excluded)
  • Temporarily changed one Join line in examples/iql/33_meta/26_debug_complex.iql.out, then ran make plan-snapshots (FAIL on 26, make exit 2); restored with git checkout
  • ./scripts/run_snapshot_tests.sh --skip-build --profile dev -j 1 -f &#39;33_meta/99_renamed_debug\.iql$&#39; (exit 1, 'No tests match filter')
  • ./scripts/run_snapshot_tests.sh --skip-build --profile nosuch ... (exit 1, binaries not found under target/nosuch)
  • Loaded .github/workflows/ci.yml with PyYAML and checked that the test job runs make plan-snapshots after make unit-test
  • Checked the 26_debug_complex snapshot for cross-product joins Join[L:[], R:[]] (none found)
  • pgrep check: no server processes left running after the runs

✅ No issues found.

  • Live validation: ✅ go - 6 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Developer runs make plan-snapshots with dev binaries and the 33_meta/17, 18, 26 .debug plan specs pass ✅ pass live 01-plan-snapshots-pass.txt (3 PASS, exit 0)
plan-snapshots passes on binaries built by cargo test --workspace --all-features (exactly as CI builds them) ✅ pass live 07-plan-snapshots-all-features.txt (3 PASS)
A plan that no longer matches its snapshot (the #290 failure) fails the gate ✅ pass live 02-stale-snapshots-fail.txt (3 FAIL with plan diffs, exit 2)
The new plans are correct: the queries behind them return the right rows and 26 has no cross product ✅ pass live 04-query-results.txt; 26_debug_complex.iql.out no longer has Join[L:[], R:[]]
A filter that matches no spec fails instead of passing silently ✅ pass live 03-empty-filter-fails.txt (exit 1)
The runner without --profile still uses target/release binaries ✅ pass live 05-default-profile.txt
The CI fast gate runs plan-snapshots right after unit-test, and editing the runner script triggers the Rust jobs ⏸️ untested no GitHub Actions cannot run locally (no act runner or GitHub runner here), so the workflow YAML was only parsed to confirm the step order and path filter. The real CI run belongs to the CI phase on the…
  • cargo build --bin inputlayer-client --bin inputlayer-server (dev profile)
  • make plan-snapshots against the current snapshots: 3/3 PASS, exit 0
  • Restored pre-#282 snapshots (git show c550f359^:examples/iql/33_meta/{17,18,26}*.iql.out) and ran make plan-snapshots: 3/3 FAIL with plan diffs, exit 2; snapshots then restored
  • Temporary spec run with ./scripts/run_snapshot_tests.sh --skip-build --profile dev -u -f 97_zz_tmp_plan_results ran the three spec queries without .debug against the live server; results checked, temp files removed
  • ./scripts/run_snapshot_tests.sh --skip-build --profile dev -f &#39;33_meta/99_does_not_exist\.iql$&#39;: 'No tests match filter', exit 1
  • ./scripts/run_snapshot_tests.sh --skip-build -f 17_debug_join (default profile) resolves to target/release binaries
  • cargo test --workspace --all-features --no-run (the CI build), then make plan-snapshots: 3/3 PASS
  • Parsed .github/workflows/ci.yml with PyYAML: the 'Run query-plan snapshots' step runs right after make unit-test in the test job, and the path filter includes scripts/run_snapshot_tests.sh

✅ No issues found.

  • Live validation: ✅ go - 4 of 5 scenarios driven live against the product
Scenario Result Live Evidence
Developer runs make plan-snapshots: specs 33_meta/17, 18 and 26 run against the real server and all pass ✅ pass live plan-snapshots-pass.log: PASS 17_debug_join, 18_debug_recursive, 26_debug_complex; 3 passed, exit 0
Adversarial: plan output no longer matches a snapshot, so make plan-snapshots fails ✅ pass live plan-snapshots-drift-detected.log: FAIL 26_debug_complex with a diff on the Join line, make Error 1
Adversarial: a filter that matches no specs (such as a renamed spec) fails instead of passing ✅ pass live empty-filter.log: 'No tests match filter', exit 1
CI order: binaries built by cargo test --workspace --all-features, then make plan-snapshots --skip-build reuses them and passes ✅ pass live ci-sequence-plan-snapshots.log: 3 passed after deleting the debug binaries and rebuilding them with cargo test --no-run
The PR workflow runs the plan-snapshots step after the unit tests in the test job, and triggers on changes to the snapshot script ⏸️ untested no The prior payload did not establish a live result. This scenario can only run live on GitHub Actions, which a PR triggers, and the CI phase owns that. The only check was a static parse of the workflow…
  • cargo build --bin inputlayer-client --bin inputlayer-server (dev profile)
  • make plan-snapshots against current snapshots: 3/3 passed, exit 0
  • Edited 26_debug_complex.iql.out (Join→JOINX) in place, then make plan-snapshots: 26 failed and the diff was shown, make exited 2; file restored afterwards
  • ./scripts/run_snapshot_tests.sh --skip-build --profile dev -j 1 -f &#39;33_meta/99_nope\.iql$&#39;: exit 1, 'No tests match filter'
  • ./scripts/run_snapshot_tests.sh --skip-build --profile dev -j 4 -f &#39;33_meta/.*debug.*\.iql$&#39;: all 7 debug specs passed
  • Simulated CI order: deleted the debug binaries, ran cargo test --workspace --all-features --no-run (the build step of make unit-test), then make plan-snapshots: 3/3 passed
  • Parsed .github/workflows/ci.yml with PyYAML: the plan-snapshots step follows unit-test in the test job, and scripts/run_snapshot_tests.sh is in the rust path filter
✅ **Document** - passed

✅ No issues found.

✅ No issues found.

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ No issues found.

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

✅ No issues found.

✅ No issues found.

@jsam jsam changed the title test(ci): gate the .debug query-plan snapshots on every PR ci: gate the .debug query-plan snapshots on every PR Oct 5, 2026
@jsam
jsam force-pushed the fm/fix-snapshot-specs branch from c4d4950 to 2fdf349 Compare October 5, 2026 17:51
@jsam jsam changed the title ci: gate the .debug query-plan snapshots on every PR ci: gate the .debug query-plan snapshot specs on every PR Oct 5, 2026
@jsam
jsam marked this pull request as ready for review October 5, 2026 18:01
@jsam
test(ci): gate the .debug query-plan snapshots on every PR
f07a293 
The three plan specs that #254's join-planner change broke
(33_meta/17_debug_join, 18_debug_recursive, 26_debug_complex) were
regenerated by #282 and pass on main. The new plans are correct: joins
follow body order, the identity Maps are gone, and 26's last rule no
longer builds a cross product (Join[L:[], R:[]]) before joining on PId.

They still ran only in the full suite, so a plan change could land
unnoticed. The fast gate's test job now runs them (make plan-snapshots)
with the dev binaries cargo test already built, so no second build.

- run_snapshot_tests.sh: --profile picks the binaries' cargo profile
  (default release); a filter that matches no spec fails instead of
  passing vacuously.
- ci.yml path filter includes the runner script.

Closes #290
@jsam jsam mentioned this pull request Oct 5, 2026
Merged
@jsam
jsam force-pushed the fm/fix-snapshot-specs branch from 2fdf349 to f07a293 Compare October 5, 2026 18:19
@jsam

jsam commented Oct 5, 2026

Copy link
Copy Markdown
Contributor Author

Closing as superseded: the three specs already pass on main, and the new PR gate (#370) runs the whole spec corpus on every PR. The two useful script changes (fail when a -f filter matches no spec; --profile dev) move into #370.

@jsam jsam closed this Oct 5, 2026
jsam added a commit that referenced this pull request Oct 5, 2026
@jsam
test(snapshots): fail a filter that matches no spec; --profile NAME (…
2c891d2 
…from #362)

Folds #362's runner changes into the PR gate: a -f filter (or an
--affected category set) that matches nothing fails instead of passing
vacuously, so a renamed spec cannot drop out of its gate, and --profile
picks the binaries' cargo profile (dev = target/debug; --debug is its
shorthand).
jsam added a commit that referenced this pull request Oct 5, 2026
@jsam
test(snapshots): fail a filter that matches no spec; --profile NAME (…
f8f51d2 
…from #362)

Folds #362's runner changes into the PR gate: a -f filter (or an
--affected category set) that matches nothing fails instead of passing
vacuously, so a renamed spec cannot drop out of its gate, and --profile
picks the binaries' cargo profile (dev = target/debug; --debug is its
shorthand).
jsam added a commit that referenced this pull request Oct 5, 2026
@jsam
test(snapshots): fail a filter that matches no spec; --profile NAME (…
f1d97f0 
…from #362)

Folds #362's runner changes into the PR gate: a -f filter (or an
--affected category set) that matches nothing fails instead of passing
vacuously, so a renamed spec cannot drop out of its gate, and --profile
picks the binaries' cargo profile (dev = target/debug; --debug is its
shorthand).
jsam added a commit that referenced this pull request Oct 5, 2026
@jsam
ci: PR gate runs scenarios, affected specs and the oracle from one de…
32b56c6 
…bug build; add main.yml on push to main (#370)

* ci: PR gate runs scenarios, affected specs and the oracle from one debug build; main.yml on push to main (#339)

- Tests job: make unit-test (unit, integration, scenario and oracle tests),
  then the snapshot specs the PR affects against the target/debug binaries
  that build left (run_snapshot_tests.sh --debug --skip-build --affected),
  then the per-binary timing table in the job summary.
- run_snapshot_tests.sh: --debug (target/debug binaries, dev-profile build)
  and --affected REF. The file-to-category map moves here from
  test-affected.sh, which now delegates; leaf modules keep a narrow
  mapping and every other engine change runs the whole corpus (about
  1.5 min on 4 cores in debug), so an unmapped module costs time, not
  coverage.
- Both SDK jobs' filters add ws-protocol/, docs/spec/ and src/protocol/.
- main.yml on push to main: every spec, oracle at 50 seeds, js-test-live,
  python-sdk-live, python-test-live and release e2e-reactive with samples.
- CONTRIBUTING and TESTING.md describe the three tiers.

* test(snapshots): fail a filter that matches no spec; --profile NAME (from #362)

Folds #362's runner changes into the PR gate: a -f filter (or an
--affected category set) that matches nothing fails instead of passing
vacuously, so a renamed spec cannot drop out of its gate, and --profile
picks the binaries' cargo profile (dev = target/debug; --debug is its
shorthand).

* no-mistakes(review): Tighten affected-spec mapping, drop --profile and main.yml dispatch

* no-mistakes(document): List hnsw_index_tests.rs in TESTING.md affected-spec map

* no-mistakes(ci): The "Detect changes" check was cancelled, not failed. The operator traced it to runner starvation: GitHub-hosted ubuntu-latest jobs waited more than 15 minutes for a runner and were cancelled. The invariant is that every job in the PR gate runs on blacksmith-4vcpu-ubuntu-2404, so none of them waits on the hosted pool. In .github/workflows/ci.yml, the four jobs still on `runs-on: ubuntu-latest` now use `runs-on: blacksmith-4vcpu-ubuntu-2404`: changes (Detect changes), the supply-chain job, the Python SDK job and the JS SDK job. Every runs-on in ci.yml is now blacksmith-4vcpu-ubuntu-2404. Other workflows are outside the PR gate and were left alone. The file still parses as YAML. No docs mention ubuntu-latest, so none needed updating. The PR-description owner should add: "PR gate jobs all run on blacksmith-4vcpu, because hosted ubuntu-latest runners starved and the jobs were cancelled
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

See More

Successfully merging this pull request may close these issues.

1 participant

@jsam