Skip to content

ci: PR gate runs scenarios, affected specs and the oracle from one debug build; add main.yml on push to main - #370

Merged
jsam merged 5 commits into
mainfrom
fm/test-t05
Oct 5, 2026
Merged

jsam merged 5 commits into
mainfrom
fm/test-t05

Conversation

@jsam

@jsam jsam commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Closes #290: the .debug plan specs it names now run on every PR that touches the engine (the PR gate runs the whole spec corpus for any engine change). #362 was closed as superseded. Its empty-filter guard is folded in here, applied only to explicit -f filters.

PR gate runners: every PR-gate job now runs on blacksmith-4vcpu-ubuntu-2404. The hosted ubuntu-latest jobs (Detect changes, supply chain, and the Python and JS SDK jobs) waited 15+ minutes for a free runner and were cancelled, which held up the whole gate.

Timing on the final head (Tests, with #358's scenarios now included): 10m04 job wall. The 3 GB cargo cache downloaded at about 40 MB/s on this run, versus about 610 MB/s on the earlier run, so the restore took 2m36 instead of 34 s. Cargo test took 6m14 (1m31 compile; 95 binaries, 4,387 tests; scenarios 12 s, oracle 19 s), and the full spec corpus took 59 s. With a normal cache restore the job is about 8 minutes, against the 10-minute budget.


Intent

please leave this query batching functionliaty just refactor things so its clear and optimized for this ... auto materialize needs to be default mode of operation where rules get deployed as views and everything else is a query .. make the docs true with the correct systems design ... this is not accceptable state for production readiness (Context: implement #339 exactly as written.)

Issue #339, "Testing: PR gate restructuring and main.yml (T-05)", as written: Part of the test strategy (captain, 2026-10-05: scenarios that test the changes together; cleaner, faster suites; TESTING.md). Epic #305. Scope: ci.yml: Tests job runs cargo test --workspace --all-features (now including scenarios), then scripts/run_snapshot_tests.sh --debug --affected <merge-base>, then prints the per-binary timing table; new JS SDK job (path filter adds ws-protocol/**, docs/spec/**, src/protocol/** to both SDK jobs); new .github/workflows/main.yml on push to main: full spec corpus, oracle 50 seeds, js-test-live, python-sdk-live, python-test-live, release e2e-reactive. run_snapshot_tests.sh --debug uses target/debug binaries. Acceptance: a PR touching src/ finishes Tests in under 10 min wall on blacksmith-4vcpu; the three #290 specs fail the first main.yml run (then are fixed by #290); a PR touching only packages/inputlayer-js runs the JS job and nothing Rust.

What Changed

  • PR gate (ci.yml): the Tests (fast) job is now Tests. It makes one debug build and runs make unit-test (unit, integration, scenario and oracle tests), with the output saved to a log. It then runs run_snapshot_tests.sh --debug --skip-build --affected HEAD^1 against the target/debug binaries, and it adds a per-binary timing table to the job summary using the new scripts/test-timings.sh. Both SDK jobs now also run when ws-protocol/**, docs/spec/** or src/protocol/** change. The Rust path filter now also covers the two scripts.

  • New .github/workflows/main.yml: runs on every push to main. It runs every snapshot spec, the differential oracle at 50 seeds, the JS and Python SDK live suites (js-test-live, python-sdk-live, python-test-live) and e2e-reactive in release.

  • run_snapshot_tests.sh gains three changes:

    • --debug builds and runs the target/debug binaries.
    • --affected REF holds the file-to-category map. Leaf modules such as the vector/HNSW and temporal code map to specific categories, a changed spec runs its own category, and any other engine source, manifest, protocol or ontology-client change runs every spec.
    • A -f filter that matches no spec now fails the run instead of passing with nothing tested.

    test-affected.sh now just calls --affected. CONTRIBUTING, TESTING.md and perf-gate/README.md describe the three CI tiers: PR gate, main and full suite.

Risk Assessment

✅ Low: The fix round applies exactly the four operator decisions. It removes --profile and keeps --debug, which sets the dev profile and target/debug. It drops workflow_dispatch from main.yml and updates the CONTRIBUTING text. It adds ontology-client/* to the every-spec arm and to the TESTING.md mapping. In --affected mode it keeps only categories that still have runnable specs, using the same find criteria as the runner, and exits 0 when none remain, while an explicit -f that matches nothing still fails. Traced cases (a deleted category, a category with only _pending_ specs, an ontology-client-only change, the AFFECTED_ALL path) all behave correctly, and no new defects were found.

Testing

I built the debug server and client in an isolated target dir and drove run_snapshot_tests.sh from a throwaway clone. The full spec corpus passed on the debug binaries alone (1152 passed, 0 failed, about 74s). --affected picked exactly the expected categories: 29_temporal for a temporal_ops edit, 31_lsh for a spec edit (where it caught the broken spec), and every category for an ontology-client edit. It exited 0 for a JS-only change, a deleted category and a pending-only category. An explicit no-match -f, a bad ref, and --affected combined with -f each fail clearly. Without --debug the script still uses release. test-timings.sh renders a correct markdown table. I parsed the workflow YAML and evaluated the path filters statically. That is not a live run, so the workflow trigger and routing scenario stays untested. Two things need a real GitHub Actions run and could not be done locally: the workflow behavior, including the under-10-minute wall time on blacksmith-4vcpu, and the live SDK and e2e jobs in main.yml. These are CLI and CI changes with nothing visual, so the evidence is CLI transcripts. The clone and the build output have been removed.

  • Live validation: ✅ go - 9 of 11 scenarios driven live against the product
Scenario Result Live Evidence
With --debug, the full corpus runs against target/debug binaries and passes ✅ pass live full-corpus-debug.txt (only a debug/ target existed; 1152 passed, 0 failed, about 74s)
Without --debug the runner still uses target/release ✅ pass live affected-live-runs.txt: looks for /tmp/nm-01M46-target/release/inputlayer-*
--affected for a leaf-module edit (temporal_ops.rs) runs only 29_temporal ✅ pass live affected-live-runs.txt: 19 specs in 29_temporal, all passed
--affected for a spec edit runs only that category, and a broken spec fails the gate ✅ pass live affected-spec-change.txt: only 31_lsh ran; the edited spec failed with exit 1
--affected for an ontology-client edit selects every category ✅ pass live affected-live-runs.txt: 'affect every category - running all snapshot tests'
--affected for a JS-only change, a deleted category, or a pending-only category exits 0 without running specs ✅ pass live affected-early-exit.txt: three cases, each exit=0 'No runnable specs affected'
An explicit -f filter that matches no spec fails the run ✅ pass live affected-live-runs.txt: 'No tests match filter: no_such_spec_zzz' exit=1
--affected rejects a bad ref and a combination with -f ✅ pass live affected-early-exit.txt
test-timings.sh turns a cargo test log into a per-binary table, slowest first, with totals ✅ pass live test-timings-synthetic.md
main.yml triggers only on push to main and runs specs/oracle, JS live, Python live and e2e-reactive; a JS-only PR reaches only the JS job, and protocol paths reach both SDK jobs ⏸️ untested no The prior payload did not establish a live result. GitHub Actions cannot run locally, and the workflows were only checked statically: a PyYAML parse plus glob evaluation of the path filters. A real pu…
A PR touching src/ finishes Tests in under 10 minutes on blacksmith-4vcpu ⏸️ untested no This needs a real GitHub Actions run on the blacksmith-4vcpu runner. The local host has 32 cores and a different cache, so local timing says nothing about the budget. Remote CI on the PR measures it.
Evidence: Full spec corpus on debug binaries
target dir contents: debug
Running 1155 tests in parallel (32 workers), 4 sequentially...
Running 4 sequential tests (global state)...
  Summary
Passed:  1152
Failed:  0
Skipped: 7
All tests passed!
exit=0 wall=74s
Evidence: --affected runs: temporal, ontology-client, explicit -f no-match, default release
\### Explicit -f matching no spec
Skipping build (--skip-build)...
Starting server...
Server log: /tmp/tmp.fmECzTNE6Y/server.log
Server started (PID 1006821)
Pre-cleaning stale knowledge graphs...

========================================
  InputLayer Snapshot Tests (parallel: 32 jobs)
========================================

No tests match filter: no_such_spec_zzz
exit=1

\### Edit src/temporal_ops.rs -> only 29_temporal, debug binaries
Changes since HEAD affect these categories:
  - 29_temporal

Skipping build (--skip-build)...
Starting server...
Server log: /tmp/tmp.GUFDlSc61K/server.log
Server started (PID 1007569)
Pre-cleaning stale knowledge graphs...

========================================
  InputLayer Snapshot Tests (parallel: 32 jobs)
========================================

Running 19 tests in parallel (32 workers), 0 sequentially...


========================================
  Summary
========================================
Passed:  19
Failed:  0
Skipped: 0

All tests passed!
exit=0

\### Edit a spec in 16_vectors + ontology-client -> every category
Changes since HEAD affect every category - running all snapshot tests.

Skipping build (--skip-build)...
\### Default (no --debug) looks for release binaries
Skipping build (--skip-build)...
Binaries not found after build!
  Client: /tmp/nm-01M46-target/release/inputlayer-client
  Server: /tmp/nm-01M46-target/release/inputlayer-server
exit=1
Evidence: --affected spec-only change runs only its category
\### Spec-only change in 31_lsh -> category 31_lsh only
examples/iql/31_lsh/01_lsh_bucket.iql
Changes since HEAD affect these categories:
  - 31_lsh

Skipping build (--skip-build)...
Starting server...
Server log: /tmp/tmp.j40LR6Acbi/server.log
Server started (PID 1014229)
Pre-cleaning stale knowledge graphs...

========================================
  InputLayer Snapshot Tests (parallel: 32 jobs)
========================================

Running 5 tests in parallel (32 workers), 0 sequentially...

FAIL [31_lsh] 01_lsh_bucket.iql
  --- Diff (expected vs actual) ---
39a40,41
> > % touched
> Parse error: Unrecognized statement: % touched


========================================
  Summary
========================================
Passed:  4
Failed:  1
Skipped: 0

Some tests failed!
Run with -v for detailed output

=== Server Log (last 50 lines) ===
^[[2m2026-10-05T20:20:14.117588Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_job_complete_messages ^[[3mprogram_len^[[0m^[[2m=^[[0m15 ^[[3mtotal_ms^[[0m^[[2m=^[[0m0
^[[2m2026-10-05T20:20:14.117648Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::rest::handlers::ws^[[0m^[[2m:^[[0m ws_session_disconnecting ^[[3mactive_sessions^[[0m^[[2m=^[[0m4
^[[2m2026-10-05T20:20:14.117690Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_complete ^[[3mprogram_len^[[0m^[[2m=^[[0m15 ^[[3mcompute_ms^[[0m^[[2m=^[[0m0 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m13 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m56^[[0m
^[[2m2026-10-05T20:20:14.117745Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::rest::handlers::ws::execute^[[0m^[[2m:^[[0m ws_execute_end ^[[3mprogram_len^[[0m^[[2m=^[[0m15 ^[[3melapsed_ms^[[0m^[[2m=^[[0m0 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m13 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m56^[[0m
^[[2m2026-10-05T20:20:14.118399Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::rest::handlers::ws::execute^[[0m^[[2m:^[[0m ws_execute_start ^[[3mprogram_len^[[0m^[[2m=^[[0m14 ^[[3mparam_count^[[0m^[[2m=^[[0m0 ^[[3mprogram_preview^[[0m^[[2m=^[[0m.session clear ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m14 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m55^[[0m
^[[2m2026-10-05T20:20:14.118530Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::rest::handlers::ws::execute^[[0m^[[2m:^[[0m ws_execute_end ^[[3mprogram_len^[[0m^[[2m=^[[0m14 ^[[3melapsed_ms^[[0m^[[2m=^[[0m0 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m14 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m55^[[0m
^[[2m2026-10-05T20:20:14.119054Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::rest::handlers::ws::execute^[[0m^[[2m:^[[0m ws_execute_start ^[[3mprogram_len^[[0m^[[2m=^[[0m35 ^[[3mparam_count^[[0m^[[2m=^[[0m0 ^[[3mprogram_preview^[[0m^[[2m=^[[0m.kg drop test_lsh_similarity_n31t04 ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m15 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m76^[[0m
^[[2m2026-10-05T20:20:14.119207Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_job_start ^[[3mprogram_len^[[0m^[[2m=^[[0m35
^[[2m2026-10-05T20:20:14.119242Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::storage_engine^[[0m^[[2m:^[[0m kg_ensure_exists ^[[3mkg^[[0m^[[2m=^[[0mdefault ^[[3melapsed_ms^[[0m^[[2m=^[[0m0
^[[2m2026-10-05T20:20:14.119278Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_parse_complete ^[[3mprogram_len^[[0m^[[2m=^[[0m35 ^[[3mstatements^[[0m^[[2m=^[[0m1 ^[[3mparse_ms^[[0m^[[2m=^[[0m0
^[[2m2026-10-05T20:20:14.119307Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m meta_kg_drop_start ^[[3mkg^[[0m^[[2m=^[[0mtest_lsh_similarity_n31t04
^[[2m2026-10-05T20:20:14.120085Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::storage_engine^[[0m^[[2m:^[[0m kg_drop_finish_complete ^[[3mkg^[[0m^[[2m=^[[0mtest_lsh_probes_n31t02 ^[[3melapsed_ms^[[0m^[[2m=^[[0m45
^[[2m2026-10-05T20:20:14.120114Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m meta_kg_drop_finish_ok ^[[3mkg^[[0m^[[2m=^[[0mtest_lsh_probes_n31t02
^[[2m2026-10-05T20:20:14.120177Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_statement_exec_complete ^[[3mprogram_len^[[0m^[[2m=^[[0m31 ^[[3mstmt_exec_ms^[[0m^[[2m=^[[0m61 ^[[3msession_facts^[[0m^[[2m=^[[0m0 ^[[3msession_rules^[[0m^[[2m=^[[0m0
^[[2m2026-10-05T20:20:14.120199Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_job_complete_messages ^[[3mprogram_len^[[0m^[[2m=^[[0m31 ^[[3mtotal_ms^[[0m^[[2m=^[[0m62
^[[2m2026-10-05T20:20:14.120338Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_complete ^[[3mprogram_len^[[0m^[[2m=^[[0m31 ^[[3mcompute_ms^[[0m^[[2m=^[[0m62 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m7 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m71^[[0m
^[[2m2026-10-05T20:20:14.127402Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::storage_engine^[[0m^[[2m:^[[0m save_metadata_complete ^[[3melapsed_ms^[[0m^[[2m=^[[0m3
^[[2m2026-10-05T20:20:14.127436Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::storage_engine^[[0m^[[2m:^[[0m kg_drop_prepare_complete ^[[3mkg^[[0m^[[2m=^[[0mtest_lsh_similarity_n31t04 ^[[3melapsed_ms^[[0m^[[2m=^[[0m8
^[[2m2026-10-05T20:20:14.127448Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m meta_kg_drop_prepare_ok ^[[3mkg^[[0m^[[2m=^[[0mtest_lsh_similarity_n31t04
^[[2m2026-10-05T20:20:14.128562Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::rest::handlers::ws::execute^[[0m^[[2m:^[[0m ws_execute_end ^[[3mprogram_len^[[0m^[[2m=^[[0m31 ^[[3melapsed_ms^[[0m^[[2m=^[[0m70 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m7 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m71^[[0m
^[[2m2026-10-05T20:20:14.129397Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::rest::handlers::ws^[[0m^[[2m:^[[0m ws_session_disconnecting ^[[3mactive_sessions^[[0m^[[2m=^[[0m3
^[[2m2026-10-05T20:20:14.136214Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::storage_engine^[[0m^[[2m:^[[0m save_metadata_complete ^[[3melapsed_ms^[[0m^[[2m=^[[0m6
^[[2m2026-10-05T20:20:14.140424Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::storage_engine^[[0m^[[2m:^[[0m kg_drop_finish_complete ^[[3mkg^[[0m^[[2m=^[[0mtest_lsh_bucket_n31t01 ^[[3melapsed_ms^[[0m^[[2m=^[[0m26
^[[2m2026-10-05T20:20:14.140931Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m meta_kg_drop_finish_ok ^[[3mkg^[[0m^[[2m=^[[0mtest_lsh_bucket_n31t01
^[[2m2026-10-05T20:20:14.141024Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_statement_exec_complete ^[[3mprogram_len^[[0m^[[2m=^[[0m31 ^[[3mstmt_exec_ms^[[0m^[[2m=^[[0m59 ^[[3msession_facts^[[0m^[[2m=^[[0m0 ^[[3msession_rules^[[0m^[[2m=^[[0m0
^[[2m2026-10-05T20:20:14.141050Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_job_complete_messages ^[[3mprogram_len^[[0m^[[2m=^[[0m31 ^[[3mtotal_ms^[[0m^[[2m=^[[0m60
^[[2m2026-10-05T20:20:14.141274Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_complete ^[[3mprogram_len^[[0m^[[2m=^[[0m31 ^[[3mcompute_ms^[[0m^[[2m=^[[0m60 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m12 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m72^[[0m
^[[2m2026-10-05T20:20:14.142614Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::rest::handlers::ws::execute^[[0m^[[2m:^[[0m ws_execute_end ^[[3mprogram_len^[[0m^[[2m=^[[0m31 ^[[3melapsed_ms^[[0m^[[2m=^[[0m61 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m12 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m72^[[0m
^[[2m2026-10-05T20:20:14.144127Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::rest::handlers::ws^[[0m^[[2m:^[[0m ws_session_disconnecting ^[[3mactive_sessions^[[0m^[[2m=^[[0m2
^[[2m2026-10-05T20:20:14.146382Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::storage_engine^[[0m^[[2m:^[[0m save_metadata_complete ^[[3melapsed_ms^[[0m^[[2m=^[[0m2
^[[2m2026-10-05T20:20:14.148704Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::storage_engine^[[0m^[[2m:^[[0m kg_drop_finish_complete ^[[3mkg^[[0m^[[2m=^[[0mtest_lsh_similarity_n31t04 ^[[3melapsed_ms^[[0m^[[2m=^[[0m21
^[[2m2026-10-05T20:20:14.148735Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m meta_kg_drop_finish_ok ^[[3mkg^[[0m^[[2m=^[[0mtest_lsh_similarity_n31t04
^[[2m2026-10-05T20:20:14.148792Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_statement_exec_complete ^[[3mprogram_len^[[0m^[[2m=^[[0m35 ^[[3mstmt_exec_ms^[[0m^[[2m=^[[0m29 ^[[3msession_facts^[[0m^[[2m=^[[0m0 ^[[3msession_rules^[[0m^[[2m=^[[0m0
^[[2m2026-10-05T20:20:14.148838Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_job_complete_messages ^[[3mprogram_len^[[0m^[[2m=^[[0m35 ^[[3mtotal_ms^[[0m^[[2m=^[[0m29
^[[2m2026-10-05T20:20:14.148994Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_complete ^[[3mprogram_len^[[0m^[[2m=^[[0m35 ^[[3mcompute_ms^[[0m^[[2m=^[[0m29 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m15 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m76^[[0m
^[[2m2026-10-05T20:20:14.149123Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::rest::handlers::ws::execute^[[0m^[[2m:^[[0m ws_execute_end ^[[3mprogram_len^[[0m^[[2m=^[[0m35 ^[[3melapsed_ms^[[0m^[[2m=^[[0m30 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m15 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m76^[[0m
^[[2m2026-10-05T20:20:14.149635Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::rest::handlers::ws^[[0m^[[2m:^[[0m ws_session_disconnecting ^[[3mactive_sessions^[[0m^[[2m=^[[0m1
^[[2m2026-10-05T20:20:15.264596Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::rest::handlers::ws^[[0m^[[2m:^[[0m ws_connection_start ^[[3mkg^[[0m^[[2m=^[[0mdefault
^[[2m2026-10-05T20:20:15.264860Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m audit_auth_apikey_success ^[[3musername^[[0m^[[2m=^[[0m"admin" ^[[3mcredential^[[0m^[[2m=^[[0mapikey:bootstrap#3
^[[2m2026-10-05T20:20:15.264904Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::rest::handlers::ws^[[0m^[[2m:^[[0m ws_authenticated ^[[3mkg^[[0m^[[2m=^[[0mdefault ^[[3musername^[[0m^[[2m=^[[0madmin ^[[3mrole^[[0m^[[2m=^[[0madmin ^[[3mcredential^[[0m^[[2m=^[[0mapikey:bootstrap#3
^[[2m2026-10-05T20:20:15.264932Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::storage_engine^[[0m^[[2m:^[[0m kg_ensure_exists ^[[3mkg^[[0m^[[2m=^[[0mdefault ^[[3melapsed_ms^[[0m^[[2m=^[[0m0
^[[2m2026-10-05T20:20:15.264986Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::rest::handlers::ws^[[0m^[[2m:^[[0m ws_session_created ^[[3mkg^[[0m^[[2m=^[[0mdefault ^[[3mactive_sessions^[[0m^[[2m=^[[0m1
^[[2m2026-10-05T20:20:15.266639Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::rest::handlers::ws::execute^[[0m^[[2m:^[[0m ws_execute_start ^[[3mprogram_len^[[0m^[[2m=^[[0m8 ^[[3mparam_count^[[0m^[[2m=^[[0m0 ^[[3mprogram_preview^[[0m^[[2m=^[[0m.kg list ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m1 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m48^[[0m
^[[2m2026-10-05T20:20:15.266897Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_job_start ^[[3mprogram_len^[[0m^[[2m=^[[0m8
^[[2m2026-10-05T20:20:15.266981Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::storage_engine^[[0m^[[2m:^[[0m kg_ensure_exists ^[[3mkg^[[0m^[[2m=^[[0mdefault ^[[3melapsed_ms^[[0m^[[2m=^[[0m0
^[[2m2026-10-05T20:20:15.267038Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_parse_complete ^[[3mprogram_len^[[0m^[[2m=^[[0m8 ^[[3mstatements^[[0m^[[2m=^[[0m1 ^[[3mparse_ms^[[0m^[[2m=^[[0m0
^[[2m2026-10-05T20:20:15.267125Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_job_complete_messages ^[[3mprogram_len^[[0m^[[2m=^[[0m8 ^[[3mtotal_ms^[[0m^[[2m=^[[0m0
^[[2m2026-10-05T20:20:15.267283Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::handler^[[0m^[[2m:^[[0m query_complete ^[[3mprogram_len^[[0m^[[2m=^[[0m8 ^[[3mcompute_ms^[[0m^[[2m=^[[0m0 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m1 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m48^[[0m
^[[2m2026-10-05T20:20:15.267326Z^[[0m ^[[32m INFO^[[0m ^[[1mws_request^[[0m: ^[[2minputlayer::protocol::rest::handlers::ws::execute^[[0m^[[2m:^[[0m ws_execute_end ^[[3mprogram_len^[[0m^[[2m=^[[0m8 ^[[3melapsed_ms^[[0m^[[2m=^[[0m0 ^[[3mok^[[0m^[[2m=^[[0mtrue ^[[2m^[[3mrequest_id^[[0m^[[2m=^[[0m1 ^[[3mmsg_bytes^[[0m^[[2m=^[[0m48^[[0m
^[[2m2026-10-05T20:20:15.267896Z^[[0m ^[[32m INFO^[[0m ^[[2minputlayer::protocol::rest::handlers::ws^[[0m^[[2m:^[[0m ws_session_disconnecting ^[[3mactive_sessions^[[0m^[[2m=^[[0m1
=== End Server Log ===
exit=1
server process during run:
/bin/bash -c source ~/.claude/shell-snapshots/snapshot-bash-1791231520389-nhvbuk.sh 2>/dev/null || true && shopt -u extglob 2>/dev/null || true && { \builtin unalias -- 'unsetenv'; \builtin unset -f -- 'unsetenv'; } >/dev/null 2>&1 || true && eval 'cd /tmp/nm-01M46-clone; E=~/.no-mistakes/evidence/01M46V6JVZM1PP491M4FD44SHW; export CARGO_TARGET_DIR=/tmp/nm-01M46-target INPUTLAYER_TEST_PORT=18931 { echo "### Spec-only change in 31_lsh -> category 31_lsh only" f=$(ls examples/iql/31_lsh/*.iql | head -1); echo "% touched" >> "$f"; git diff --name-only HEAD ( ./scripts/run_snapshot_tests.sh --debug --skip-build --affected HEAD; echo "exit=$?" ) &  sleep 4; echo "server process during run:"; ps -eo args | grep "[i]nputlayer-server" | grep 18931 -m1 || ps -eo args | grep "[n]m-01M46-target/debug/inputlayer-server" | head -1 wait; git checkout -q -- . } 2>&1 | tee $E/affected-spec-change.txt' < /dev/null && pwd -P >| /tmp/claude-4033-cwd
Evidence: --affected early exits and guards
\### JS-only change -> no specs
No runnable specs affected by the changes since HEAD.
exit=0

\### Deleted category 42 -> exit 0, nothing to run
No runnable specs affected by the changes since HEAD.
exit=0

\### New category with only a _pending_ spec -> exit 0
No runnable specs affected by the changes since HEAD.
exit=0

\### Explicit -f matching no spec -> fails
Skipping build (--skip-build)...
Binaries not found after build!
  Client: /tmp/nm-01M46-target/debug/inputlayer-client
  Server: /tmp/nm-01M46-target/debug/inputlayer-server
exit=1

\### --affected with --filter -> rejected
--affected and --filter cannot be combined
exit=1

\### --affected bad ref -> error
fatal: bad revision 'nosuchref123'
Cannot diff against nosuchref123
exit=1

\### test-affected.sh with no args and clean tree
error: invalid option: --debug
usage: git diff [<options>] [<commit>] [--] [<path>...]
   or: git diff [<options>] --cached [--merge-base] [<commit>] [--] [<path>...]
   or: git diff [<options>] [--merge-base] <commit> [<commit>...] <commit> [--] [<path>...]
   or: git diff [<options>] <commit>...<commit> [--] [<path>...]
   or: git diff [<options>] <blob> <blob>
   or: git diff [<options>] --no-index [--] <path> <path>

common diff options:
  -z            output diff-raw with lines terminated with NUL.
  -p            output patch format.
  -u            synonym for -p.
  --patch-with-raw
                output both a patch and the diff-raw format.
  --stat        show diffstat instead of patch.
  --numstat     show numeric diffstat instead of patch.
  --patch-with-stat
                output a patch and prepend its diffstat.
  --name-only   show only names of changed files.
  --name-status show names and status of changed files.
  --full-index  show full object name on index lines.
  --abbrev=<n>  abbreviate object names in diff-tree header and diff-raw.
  -R            swap input file pairs.
  -B            detect complete rewrites.
  -M            detect renames.
  -C            detect copies.
  --find-copies-harder
                try unchanged files as candidate for copy detection.
  -l<n>         limit rename attempts up to <n> paths.
  -O<file>      reorder diffs according to the <file>.
  -S<string>    find filepair whose only one side contains the string.
  --pickaxe-all
                show all files diff when -S is used and hit is found.
  -a  --text    treat all files as text.

Cannot diff against --debug
exit=1

\### --help lists --debug and --affected
Usage: ./scripts/run_snapshot_tests.sh [OPTIONS]

Options:
  -v, --verbose    Show verbose diff output on failures (forces sequential)
  -f, --filter     Only run tests matching pattern
  -u, --update     Update .out files with actual output (forces sequential)
  -j, --jobs N     Parallel jobs (default: 32, 0 or 1 = sequential)
  --skip-build     Skip cargo build
  --debug          Use the target/debug binaries cargo test builds (default: release)
  --affected REF   Only run the categories the changes since REF affect
  -h, --help       Show this help message
exit=0

\### test-affected.sh (no args, clean tree)
No runnable specs affected by the changes since HEAD.
exit=0
Evidence: Per-binary timing table
| Binary | Tests | Failed | Seconds |
|---|---:|---:|---:|
| tests/scenarios.rs (scenarios) | 4 | 1 | 12.25 |
| src/lib.rs (inputlayer) | 3 | 0 | 1.50 |
| doc-tests inputlayer | 2 | 0 | 0.30 |
| **total (3 binaries)** | 9 | 1 | 14.05 |
Evidence: Workflow triggers and path-filter checks (static)
main.yml triggers: {'push': {'branches': ['main']}}
main.yml jobs: ['specs-oracle', 'js-sdk-live', 'python-sdk-live', 'e2e-reactive']
['packages/inputlayer-js/src/client.ts'] -> ['js']
['src/protocol/handler.rs'] -> ['js', 'python', 'rust']
['ws-protocol/src/lib.rs'] -> ['js', 'python', 'rust']
['docs/spec/protocol.md'] -> ['js', 'python', 'rust']
['scripts/run_snapshot_tests.sh'] -> ['rust']
changes | needs: None | if: 
fmt | needs: changes | if: needs.changes.outputs.rust == 'true'
secrets | needs: None | if: 
lint | needs: changes | if: needs.changes.outputs.rust == 'true'
supply-chain | needs: changes | if: needs.changes.outputs.rust == 'true'
test | needs: changes | if: needs.changes.outputs.rust == 'true' && !contains(github.event.pull_request.labels.*.name, 'release') && !(startsWith(github.head_ref, 'release/') && github.event.pull_request.head.repo.full_name == github.repository)
python-sdk | needs: changes | if: needs.changes.outputs.python == 'true'
js-sdk | needs: changes | if: needs.changes.outputs.js == 'true'
test steps:
  - None :: None
  - Install Rust toolchain :: None
  - Cache cargo :: None
  - Run unit, integration, scenario and oracle tests :: make unit-test 2>&1 | tee "$RUNNER_TEMP/cargo-test.log"
  - Run affected snapshot specs (debug binaries) :: ./scripts/run_snapshot_tests.sh --debug --skip-build --affected "$(git rev-parse HEAD^1)"
  - Per-binary test times :: [ -f "$RUNNER_TEMP/cargo-test.log" ] || exit 0
./scripts/test-timings.sh "$RUNNER_TEMP/cargo-test.log" | tee -a "$GITHUB_STEP_SUMMARY"

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 5 issues found → auto-fixed ✅
  • ⚠️ scripts/run_snapshot_tests.sh:451 - The --affected map leaves out ontology-client/, so a PR that only changes that crate runs no snapshot specs, even though it is compiled into the server binary. inputlayer-ontology-client is a normal dependency in the root Cargo.toml (line 110), and src/protocol/handler.rs calls it on the server path, e.g. registry::validate_component(&#34;ontology name&#34;, name) at handler.rs:5269. Example: a PR that only edits ontology-client/src/registry.rs passes the ci.yml rust path filter (ci.yml:51), so Tests runs. affected_categories then reaches no case arm, AFFECTED is empty, and the script prints "No test-relevant changes" and exits 0. Spec coverage of that server behavior is silently skipped. This contradicts the function's own comment that 'any other change to the engine's sources or build runs the whole corpus, so a module missing from this list costs time rather than coverage', and the CONTRIBUTING:371 / TESTING.md:307 claim that every spec runs 'when the engine's sources, manifests ... change'. The same gap applies to ontology-client/Cargo.toml and ws-protocol/Cargo.toml (only the root Cargo.toml is matched; ws-protocol/* already covers the latter). Fix: add ontology-client/* to the run-all arm and list it in TESTING.md:307.
  • ⚠️ scripts/run_snapshot_tests.sh:627 - In --affected mode, a PR that deletes a whole spec category, or adds a category containing only _pending_ or _helper specs, fails the PR gate. examples/iql/*/* maps each changed path to its category directory (line 453) without checking that the directory still has runnable specs. The filter /\(23_foo\)/ then matches no file, and the new 'No tests match filter' check exits 1. This is realistic: the corpus already has gaps where categories were retired (03, 05, 23, 26, 37, 38). Example: git rm -r examples/iql/42_consistency_pack → Tests fails with 'No tests match filter: /(42_consistency_pack)/'. Smallest fix: in the --affected branch, keep only categories that still contain runnable specs and take the existing 'No test-relevant changes' exit when none remain. The no-match failure should apply only to a user-supplied -f. See also the simplification finding on that check.
  • ⚠️ scripts/run_snapshot_tests.sh:469 - Simplification: issue Testing: PR gate restructuring and main.yml (T-05) #339 asks only for run_snapshot_tests.sh --debug (use the target/debug binaries). The general --profile NAME option, with its profile-to-directory mapping and TESTING.md row, comes from ci: gate the .debug query-plan snapshot specs on every PR #362 per the commit message; neither CI workflow uses it. No Testing: PR gate restructuring and main.yml (T-05) #339 requirement needs it. Recommend removing --profile and keeping only --debug (a debug/release switch).
  • ⚠️ scripts/run_snapshot_tests.sh:625 - Simplification: failing a run whose -f filter matches no spec is ci: gate the .debug query-plan snapshot specs on every PR #362 scope (commit f1d97f0, 'from ci: gate the .debug query-plan snapshot specs on every PR #362'), not Testing: PR gate restructuring and main.yml (T-05) #339, which asks for the change to be implemented exactly as written. The check is also what turns the deleted-category case above into a gate failure. Recommend removing it from this change, or at least not applying it to the filter that --affected generates.
  • ℹ️ .github/workflows/main.yml:15 - Simplification: issue Testing: PR gate restructuring and main.yml (T-05) #339 specifies main.yml 'on push to main'. The added workflow_dispatch trigger, which runs on any ref and is documented in CONTRIBUTING, goes beyond that. Remove it unless manual runs are wanted.

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 9 of 11 scenarios driven live against the product
Scenario Result Live Evidence
With --debug, the full corpus runs against target/debug binaries and passes ✅ pass live full-corpus-debug.txt (only a debug/ target existed; 1152 passed, 0 failed, about 74s)
Without --debug the runner still uses target/release ✅ pass live affected-live-runs.txt: looks for /tmp/nm-01M46-target/release/inputlayer-*
--affected for a leaf-module edit (temporal_ops.rs) runs only 29_temporal ✅ pass live affected-live-runs.txt: 19 specs in 29_temporal, all passed
--affected for a spec edit runs only that category, and a broken spec fails the gate ✅ pass live affected-spec-change.txt: only 31_lsh ran; the edited spec failed with exit 1
--affected for an ontology-client edit selects every category ✅ pass live affected-live-runs.txt: 'affect every category - running all snapshot tests'
--affected for a JS-only change, a deleted category, or a pending-only category exits 0 without running specs ✅ pass live affected-early-exit.txt: three cases, each exit=0 'No runnable specs affected'
An explicit -f filter that matches no spec fails the run ✅ pass live affected-live-runs.txt: 'No tests match filter: no_such_spec_zzz' exit=1
--affected rejects a bad ref and a combination with -f ✅ pass live affected-early-exit.txt
test-timings.sh turns a cargo test log into a per-binary table, slowest first, with totals ✅ pass live test-timings-synthetic.md
main.yml triggers only on push to main and runs specs/oracle, JS live, Python live and e2e-reactive; a JS-only PR reaches only the JS job, and protocol paths reach both SDK jobs ⏸️ untested no The prior payload did not establish a live result. GitHub Actions cannot run locally, and the workflows were only checked statically: a PyYAML parse plus glob evaluation of the path filters. A real pu…
A PR touching src/ finishes Tests in under 10 minutes on blacksmith-4vcpu ⏸️ untested no This needs a real GitHub Actions run on the blacksmith-4vcpu runner. The local host has 32 cores and a different cache, so local timing says nothing about the budget. Remote CI on the PR measures it.
  • CARGO_TARGET_DIR=/tmp/nm-01M46-target cargo build --bin inputlayer-client --bin inputlayer-server (debug profile, isolated target dir)
  • ./scripts/run_snapshot_tests.sh --debug --skip-build (full corpus, debug only, no release build present)
  • ./scripts/run_snapshot_tests.sh --debug --skip-build --affected HEAD with src/temporal_ops.rs edited (only 29_temporal ran)
  • --affected HEAD with a spec in 31_lsh edited (only 31_lsh ran; the broken spec failed the run)
  • --affected HEAD with ontology-client/src/lib.rs edited (every category selected)
  • --affected HEAD with a JS-only change, a deleted category 42, and a new category holding only a pending spec (all exit 0 with 'No runnable specs affected')
  • ./scripts/run_snapshot_tests.sh --debug --skip-build -f no_such_spec_zzz (exit 1 'No tests match filter')
  • --affected combined with -f (rejected); --affected nosuchref123 (error exit 1)
  • ./scripts/test-affected.sh on a clean tree (delegates to --affected and exits 0)
  • ./scripts/run_snapshot_tests.sh --skip-build -f 29_temporal without --debug (looks for release binaries)
  • ./scripts/test-timings.sh on a cargo test log with passing, failing, crashed and doc-test binaries, and with no argument
  • Parsed ci.yml and main.yml with PyYAML and checked the dorny path filters against sample paths: JS-only, src/protocol, ws-protocol, docs/spec, scripts (static check only, not a live CI run)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@jsam
jsam force-pushed the fm/test-t05 branch 2 times, most recently from f5621ae to 81f8fc6 Compare October 5, 2026 18:14
@jsam jsam changed the title ci: PR gate runs scenarios, affected specs and the oracle from one debug build; JS SDK job; main.yml ci: PR gate runs scenarios, affected specs and the oracle from one debug build; main.yml on push to main Oct 5, 2026
jsam added 4 commits October 5, 2026 20:13
…bug build; main.yml on push to main (#339)

- Tests job: make unit-test (unit, integration, scenario and oracle tests),
  then the snapshot specs the PR affects against the target/debug binaries
  that build left (run_snapshot_tests.sh --debug --skip-build --affected),
  then the per-binary timing table in the job summary.
- run_snapshot_tests.sh: --debug (target/debug binaries, dev-profile build)
  and --affected REF. The file-to-category map moves here from
  test-affected.sh, which now delegates; leaf modules keep a narrow
  mapping and every other engine change runs the whole corpus (about
  1.5 min on 4 cores in debug), so an unmapped module costs time, not
  coverage.
- Both SDK jobs' filters add ws-protocol/, docs/spec/ and src/protocol/.
- main.yml on push to main: every spec, oracle at 50 seeds, js-test-live,
  python-sdk-live, python-test-live and release e2e-reactive with samples.
- CONTRIBUTING and TESTING.md describe the three tiers.
…from #362)

Folds #362's runner changes into the PR gate: a -f filter (or an
--affected category set) that matches nothing fails instead of passing
vacuously, so a renamed spec cannot drop out of its gate, and --profile
picks the binaries' cargo profile (dev = target/debug; --debug is its
shorthand).
@jsam jsam changed the title ci: PR gate runs scenarios, affected specs and the oracle from one debug build; main.yml on push to main ci: PR gate runs scenarios, affected specs and the oracle from one debug build; add main.yml on push to main Oct 5, 2026
…. The operator traced it to runner starvation: GitHub-hosted ubuntu-latest jobs waited more than 15 minutes for a runner and were cancelled. The invariant is that every job in the PR gate runs on blacksmith-4vcpu-ubuntu-2404, so none of them waits on the hosted pool. In .github/workflows/ci.yml, the four jobs still on `runs-on: ubuntu-latest` now use `runs-on: blacksmith-4vcpu-ubuntu-2404`: changes (Detect changes), the supply-chain job, the Python SDK job and the JS SDK job. Every runs-on in ci.yml is now blacksmith-4vcpu-ubuntu-2404. Other workflows are outside the PR gate and were left alone. The file still parses as YAML. No docs mention ubuntu-latest, so none needed updating. The PR-description owner should add: "PR gate jobs all run on blacksmith-4vcpu, because hosted ubuntu-latest runners starved and the jobs were cancelled
@jsam
jsam marked this pull request as ready for review October 5, 2026 20:59
@jsam
jsam merged commit 32b56c6 into main Oct 5, 2026
23 checks passed
@jsam
jsam deleted the fm/test-t05 branch October 5, 2026 21:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Snapshot specs 33_meta/17, 18, 26 (.debug plans) fail on main since #254's join-planner change

1 participant