Repository navigation
test(oracle): sustained concurrency soak with many subscribers and slow consumers - #366
Merged
Merged
Conversation
…). Both broke on the same compile error: `testkit/src/agent.rs` defined `Agent::clear_notices` twice (error E0592). **Cause:** CI builds this PR merged with main, and main's #355 added an identical `clear_notices` to `Agent`, so the merged code had two copies. The method can only be defined once. **Fix:** I merged `origin/main` into the branch (no conflicts, merge commit 562d13c) and deleted this branch's copy of `clear_notices`, keeping main's. No other duplicates came in with the merge. The soak test's other two `Agent` additions, `client_mut` and `next_any_delta`, are still there; `next_any_delta` calls `next_delta`, which now uses main's new `delta_from`. The deletion in `testkit/src/agent.rs` is not committed yet; the merge commit itself is. **Verification (local):** - `make lint` (both clippy commands with `-D warnings`): passes. - `cargo fmt --check`: passes. - `cargo test --workspace --all-features` (what `make unit-test` runs): exit 0, all 93 test binaries report `ok`
…k test was flaky: it judged too early that a consumer had finished, so it is not an engine bug. Only the two group subscribers failed. At final revision 94 they still held the results from about revision 90 (for example, 8 `reach(0, Y)` rows where the reference has none). **Rule that was broken:** a consumer may report its results as final only after it has gone a full quiet period without new pushes, counted from when the final revision was set (after the writers stopped). **Cause:** in `tests/differential_oracle/soak/consumers.rs`, the quiet period was counted from the consumer's last delta, and that delta could be older than the moment the final revision was set. The engine evaluates subscription views in the background, and the group view (all core queries, including the recursive `reach`) is the slowest. On the CI runner its last delta was already more than 500 ms old when the writers finished and the final revision was published. So the group reported final at once, while the deltas for the last few commits were still being computed. The 500 ms smoke quiet period was also below the lag CI measured (fast-subscriber p99 930 ms, stalled-subscriber max 1.47 s). **Fix:** - Both places that decide a consumer is done (`group_loop` and `subscribe_loop`) now count the quiet period from whichever is later: the last delta, or when the consumer first saw the final revision. - The smoke's default `quiet_ms` goes from 500 to 2000 ms in `tests/differential_oracle/soak/mod.rs`, which is above the lag CI measured. The field's doc comment already describes it as quiet time after the writers stop, so it now matches the code. The sustained profile already uses 3000 ms. The smoke test takes about 1.5 s longer. **Verification (local):** - `cargo fmt --check` and `make lint` (clippy with `-D warnings`) pass. - The soak test passed 3 times normally and 3 times with the server pinned to one CPU. - The full `differential_oracle` test binary passes (9/9). - I could not reproduce the CI failure locally even with the server pinned, so the fix comes from tracing the timing in the code, not from a failing local run. The changes in `consumers.rs` and `mod.rs` are not committed yet
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
I want you to really hyper focus on making the symbolic engine, you know, crisp and really truly production ready. ... just ensure that both machines are propelry utilized now (Context: implement #298 exactly as written.) Issue #298, 'Add a sustained concurrency soak with the differential oracle, many subscribers and slow consumers': From the engine production-readiness audit (2026-10-05). The differential oracle is single-threaded; there is no sustained multi-writer soak and no many-subscriber / slow-consumer-under-load coverage. Run it on the benchmark host.
What Changed
tests/differential_oracle/soak/, which runs a realinputlayer-serverunder concurrent load. Writers each own a disjoint share of the base facts, and a rule churner atomically swaps rule variants (recursion shape, negation, aggregates) while the writers run. Many consumers watch standing queries at the same time: fast and slow ones (a slow consumer has a one-frame inbox and a 4 KB socket buffer), consumers that stall, subscription groups, short-lived connections that attach to shared views, andreadauditors. A verifier thread replays the commits in revision order through the reference evaluator and checks every snapshot, delta and read at the revision it names. It also checks that every consumer ends on the reference's final state and that only slow or stalled consumers are disconnected, and only in the ways the protocol documents. Server memory is sampled during the run, and its growth is judged over the second half. By default the soak is a short smoke test that runs with the oracle.INPUTLAYER_SOAK_*variables scale it.WsClientgains paced connections (Pacing: a bounded inbox and an optionalSO_RCVBUF), plusread,subscribe_groupandping. Write replies now exposerevisionand the effective counts instatements.Agentgainsnext_any_delta,client_mutandclear_notices.EngineBuildercan now set the WS send timeout, the connection limits and the per-IP limit on unauthenticated connections, and can pin the server to CPUs withtaskset.Engine::pidreturns the server's process id.scripts/soak.shrunner, which runs the soak in release and writesresult.json,summary.mdandsettings.txttotarget/soak/latest. Addsmake soakandmake soak-remote, and a--bench soakmode inperf-gate-remote.shthat pins the soak's clients and the server to separate CPU sets on the benchmark host.TESTING.mddocuments the soak.Risk Assessment
✅ Low: The fix round fixes the admin idle-timeout failure the minimal way: it opens fresh KG-scoped connections for the final read and the health ping and closes the setup client. Every connection still open during the run already keeps itself active: consumers and groups ping every 60s, auditors read every audit_ms, writers are paced, the churner fires every churn_ms, and stalls last stall_ms (1s by default).
Testing
I ran the soak against a real inputlayer-server four ways, all through the new soak.sh and its test entry point. (1) The smoke profile passed in release; result.json, summary.md and settings.txt were written. (2) The same smoke with MAX_FAST_P99_MS=0.001 correctly reported FAIL and soak.sh exited 1. (3) A sustained-scale run with writers running 335 s, longer than the 300 s ws idle timeout, passed. It had 16 writers, 300 fast, 40 slow, 20 stalled, 20 group and 20 churn consumers, plus 4 auditors, with the server pinned to CPUs 8-31 and clients to 0-7 to mirror soak-remote's split. The oracle judged 41,609 commits; fast consumers had no disconnects, and server memory grew 115→126 MiB over the second half, within the 25% limit. The client cores averaged about 85-99% and the server cores about 70-93% busy while writers ran (the sampler measures whole cores, so other host load is included). (4) The debug test-suite smoke passed. The remote benchmark host (make soak-remote) was not driven, because it pushes to an external SSH host; the local 32-core pinned run stands in for it. I removed the transient target/soak output.
Evidence: Smoke soak transcript (PASS)
~/.no-mistakes/evidence/01M46KER2PWTCP4DA6NFJWMPXR/smoke-run)Evidence: Adversarial p99 limit transcript (FAIL, exit 1)
Evidence: Sustained-scale pinned 335s soak transcript (PASS)
~/.no-mistakes/evidence/01M46KER2PWTCP4DA6NFJWMPXR/long-pinned-run)Evidence: Per-CPU-group utilisation during sustained run
Evidence: Debug test-suite smoke
Evidence: Sustained run verdict
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 1 issue found → auto-fixed ✅
tests/differential_oracle/soak/mod.rs:376- The sustained soak will always fail, because theadminconnection sits idle for the whole run. It is opened at mod.rs:234 and used only for.kg create. After that it sends nothing until the finaladmin.read(...)at mod.rs:376 andadmin.ping()at mod.rs:405. The engine closes a WebSocket afterws_idle_timeout_ms, which defaults to 300 000 ms (src/config.rs:851). The testkit config does not override this setting. Only client text frames and replies reset the idle timer; pongs to the server's heartbeat fall into_ => {}(src/protocol/rest/handlers/ws.rs:454-504). Withsoak.sh --profile sustained(INPUTLAYER_SOAK_SECS=1800, also themake soak/make soak-remotedefault) or any run with SECS > ~300, the engine closes the admin connection at about 5 minutes. When the writers stop, the final read fails with "final read: ...". Thenfinal_revisionfalls back to 1, so every subscriber'sclaim_finalfails with StaleRevision (consumers.rs:203), and the closingadmin.ping()adds "the engine stopped answering". The run ends FAIL even when the engine behaves correctly. This failure is reachable on the intended benchmark-host run. The consumers already keep their connections alive withPING_EVERY(consumers.rs:25), but the admin connection does not. Fix: pingadminin the write-deadline wait loop (mod.rs:356-358) at least every 60 s, or open a freshWsClientfor the final read and the health ping. The unusedsetup_client(mod.rs:242) is harmless because it is never used after setup.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
./scripts/soak.sh --profile smoke(release build, real inputlayer-server)./scripts/soak.sh --profile smoke --set MAX_FAST_P99_MS=0.001(adversarial: the limit must fail the run)./scripts/soak.sh --profile sustained --secs 330 --server-cpus 8-31 --client-cpus 0-7with a /proc/stat per-CPU-group sampler running alongsidecargo test --all-features --test differential_oracle concurrent_soak_agrees_with_reference -- --nocapture(debug test-suite smoke path)✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
Benchmark-host run (sam-dev-benchmarks, 32 vCPU; server pinned 8-31, clients 0-7)
Sustained profile at 99c1cc6, 2026-10-05 18:16-18:47Z: 16 writers × 20 programs/s for 30 min, rule churn every 1 s, 16 nodes; 300 fast, 40 slow, 20 stalled, 20 group, 20 churn and 4 read consumers; engine defaults.
ws_send_timeout, 889ws_slow_subscriber_disconnected). All of them reconnected and resubscribed correctly. Before being cut off, slow consumers ran up to ~60 s behind.GLIBC_TUNABLES=glibc.malloc.arena_max=1. With subscribers and rule churn it went 64 → 98 MiB with default arenas and 37 → 46 MiB (flattening) with one arena, though one arena also cut throughput ~40%. So most of the growth is glibc's per-thread arenas keeping freed memory, not live objects; a small residual is still unexplained at that length. This is an engine follow-up (allocator choice or arena cap plus trimming), not part of this PR.