Repository navigation
fix(protocol): run shared standing-query evaluations one at a time, with at most one more pending - #355
Conversation
|
Benchmark host. The 1,000-session re-measure of the permit split (
A replay of every session's deltas against a fresh query found 0 lost, duplicated, misrouted or diverged on both arms. Any further benchmark re-run waits until the host is back. Please hold this PR. Small hosts regress. On a 4-core host the default split leaves requests one compute permit (2 permits: 1 request, 1 standing). One long query on any connection then holds every write behind it. |
… evaluating and one pending A family of parameterized views (one bound subscription per session) evaluated its lifted query once per revision a refreshing view asked for. Under a steady stream of commits every view's follow-up holds a newer snapshot than the latest round, so it started another full round: at 1,000 voice sessions, 30 to 110 full evaluations a second, each serving 2 to 8 views, all queued on the compute permits. A view waited up to 9 s for its round, and probes in perf-gate sessions arrived 11 to 16 s after their write (issue #292's "missing" and "stray" deltas were these late ones; nothing was lost, duplicated or misrouted). A round now starts pending and takes its snapshot only when it starts, after the family's previous round finished (one evaluates at a time). A view that needs a newer revision than the round evaluating joins the pending round, which is then no older than the view's snapshot, so a family evaluates one round at a time and queues at most one more, and a refresh waits for at most two rounds.
…so request bursts cannot delay deltas Every request, a one-row write included, takes a compute permit from one FIFO semaphore, and holds it while it waits for its knowledge graph's commit. Standing- query evaluations queued on the same semaphore, so under a write burst a refresh waited behind every queued write (and writes behind every queued refresh): at 1,000 sessions, receipt writes spent p50 4.2 s waiting while the WAL persist took 1 ms, and the probe's own write was 3 to 7 s old before any view could start on it. Standing-query evaluations now run under a semaphore of their own, as wide as the compute permits: a burst of writes or queries never delays a delta, and evaluations never delay a request.
…ing else 96 sessions each subscribe to their own bound query (one parameterized family, as voice-agent sessions do) while 32 writers commit receipts back to back; each receipt is in every view's dependencies but changes no result. A prober changes one session's result at a time. Every probe's delta must reach exactly its session, once, within 5 s of the write's acknowledgement; no session may get a delta no probe caused; every view keeps the agent contract (contiguous seq, increasing revision, exact inserts and retracts) and ends equal to a fresh query. The testkit agent gains poll_delta and clear_notices, so long-running agents can poll without a 60 s timeout and without keeping every change notification.
…ir own query threads
…_connection` (tests/ws_pipeline_tests.rs) relied on the host's core count. Commit 410425b splits the compute budget between requests and standing-query evaluations. On the CI runner the budget is about 2 permits, which leaves requests 1. The client's TRIANGLES query held that 1 request permit, so the other writer's `+seen(1)` queued behind it. The commit, and so the delta, came only after the long query finished. The production behavior is as the user decided; the test's setup no longer held on small hosts. It passed locally only because this machine has 32 cores. Violated invariant: the test fixture must give requests at least 2 permits so a writer can commit while a long query runs, on any host. The only place that sets this is `start_server`, which every test in the file uses. Fix: `start_server` now sets the budget explicitly with `with_compute_permits(COMPUTE_PERMITS = 4)`. The default half share gives requests 2 permits and standing queries 2, independent of the host. No production code changed. Verification: setting the budget to 2 permits (the CI case) reproduced the exact failure locally ("no delta while the query ran"). With the fixed budget of 4, `cargo test --test ws_pipeline_tests` passed 9/9 on three consecutive runs. `cargo fmt --check` and clippy are clean
…uery never blocks every write Splitting the compute budget in half left a 2-permit host (4 or 5 cores, such as the 4-vCPU CI runners) one request permit: one long query then held every write, and every delta after it, behind it. tests/e2e_reactive's agent_receives_deltas_while_its_own_long_query_runs failed pinned to 4 cores and passed on main. Requests now keep at least two permits whenever the budget has two, and standing-query evaluations at least one, so a 2-permit host runs three computations at once (one over its budget, only there); larger budgets split as before. ws_pipeline_tests now runs on the smallest split budget (2), so per-PR CI covers a request running beside a long query; it fails with the old split.
…e long query never blocks every write" This reverts commit ae275cc304eb02ee680c06eaa2475e7e4c380b55.
…s_on_its_connection` (tests/ws_pipeline_tests.rs) relied on the host's core count. Commit 410425b splits the compute budget between requests and standing-query evaluations. On the CI runner the budget is about 2 permits, which leaves requests 1. The client's TRIANGLES query held that 1 request permit, so the other writer's `+seen(1)` queued behind it. The commit, and so the delta, came only after the long query finished. The production behavior is as the user decided; the test's setup no longer held on small hosts. It passed locally only because this machine has 32 cores. Violated invariant: the test fixture must give requests at least 2 permits so a writer can commit while a long query runs, on any host. The only place that sets this is `start_server`, which every test in the file uses. Fix: `start_server` now sets the budget explicitly with `with_compute_permits(COMPUTE_PERMITS = 4)`. The default half share gives requests 2 permits and standing queries 2, independent of the host. No production code changed. Verification: setting the budget to 2 permits (the CI case) reproduced the exact failure locally ("no delta while the query ran"). With the fixed budget of 4, `cargo test --test ws_pipeline_tests` passed 9/9 on three consecutive runs. `cargo fmt --check` and clippy are clean" This reverts commit 0d50729.
…nd standing evaluations" This reverts commit 410425b.
…ir own, so request bursts cannot delay deltas" This reverts commit 21c693d.
…rt deliveries during the load
0d50729 to
a712c01
Compare
|
Hold lifted: the permit split is gone. Splitting the compute budget between requests and standing queries starved deltas on 4- and 8-core hosts at about 1,000 sessions, which main did not. This PR now keeps only one-at-a-time shared rounds per family, plus the tests and docs. Requests and refreshes share one compute budget, as on main. The net diff against main touches no
Three host sizes, same test, devbox (lab, not the benchmark host), quiet runs, rounds-only build
No run had a Voice-agent pack, 1,000 sessions, 16 server cores, same run ( |
…). Both broke on the same compile error: `testkit/src/agent.rs` defined `Agent::clear_notices` twice (error E0592). **Cause:** CI builds this PR merged with main, and main's #355 added an identical `clear_notices` to `Agent`, so the merged code had two copies. The method can only be defined once. **Fix:** I merged `origin/main` into the branch (no conflicts, merge commit 562d13c) and deleted this branch's copy of `clear_notices`, keeping main's. No other duplicates came in with the merge. The soak test's other two `Agent` additions, `client_mut` and `next_any_delta`, are still there; `next_any_delta` calls `next_delta`, which now uses main's new `delta_from`. The deletion in `testkit/src/agent.rs` is not committed yet; the merge commit itself is. **Verification (local):** - `make lint` (both clippy commands with `-D warnings`): passes. - `cargo fmt --check`: passes. - `cargo test --workspace --all-features` (what `make unit-test` runs): exit 0, all 93 test binaries report `ok`
…ow consumers (#366) * test(oracle): sustained concurrency soak judged by the reference at every revision * test(soak): soak.sh runner, make soak and soak-remote on the benchmark host * test(soak): judge memory growth over the run's second half * test(soak): raise the descriptor limit for sustained runs * no-mistakes(review): Use fresh soak connections for final read and health ping * no-mistakes(document): Note soak client CPU pinning in remote script header * no-mistakes(ci): I fixed both failing checks (Clippy and Tests (fast)). Both broke on the same compile error: `testkit/src/agent.rs` defined `Agent::clear_notices` twice (error E0592). **Cause:** CI builds this PR merged with main, and main's #355 added an identical `clear_notices` to `Agent`, so the merged code had two copies. The method can only be defined once. **Fix:** I merged `origin/main` into the branch (no conflicts, merge commit 562d13c) and deleted this branch's copy of `clear_notices`, keeping main's. No other duplicates came in with the merge. The soak test's other two `Agent` additions, `client_mut` and `next_any_delta`, are still there; `next_any_delta` calls `next_delta`, which now uses main's new `delta_from`. The deletion in `testkit/src/agent.rs` is not committed yet; the merge commit itself is. **Verification (local):** - `make lint` (both clippy commands with `-D warnings`): passes. - `cargo fmt --check`: passes. - `cargo test --workspace --all-features` (what `make unit-test` runs): exit 0, all 93 test binaries report `ok` * no-mistakes(ci): I fixed the failing Tests (fast) check. The PR's soak test was flaky: it judged too early that a consumer had finished, so it is not an engine bug. Only the two group subscribers failed. At final revision 94 they still held the results from about revision 90 (for example, 8 `reach(0, Y)` rows where the reference has none). **Rule that was broken:** a consumer may report its results as final only after it has gone a full quiet period without new pushes, counted from when the final revision was set (after the writers stopped). **Cause:** in `tests/differential_oracle/soak/consumers.rs`, the quiet period was counted from the consumer's last delta, and that delta could be older than the moment the final revision was set. The engine evaluates subscription views in the background, and the group view (all core queries, including the recursive `reach`) is the slowest. On the CI runner its last delta was already more than 500 ms old when the writers finished and the final revision was published. So the group reported final at once, while the deltas for the last few commits were still being computed. The 500 ms smoke quiet period was also below the lag CI measured (fast-subscriber p99 930 ms, stalled-subscriber max 1.47 s). **Fix:** - Both places that decide a consumer is done (`group_loop` and `subscribe_loop`) now count the quiet period from whichever is later: the last delta, or when the consumer first saw the final revision. - The smoke's default `quiet_ms` goes from 500 to 2000 ms in `tests/differential_oracle/soak/mod.rs`, which is above the lag CI measured. The field's doc comment already describes it as quiet time after the writers stop, so it now matches the code. The sustained profile already uses 3000 ms. The smoke test takes about 1.5 s longer. **Verification (local):** - `cargo fmt --check` and `make lint` (clippy with `-D warnings`) pass. - The soak test passed 3 times normally and 3 times with the server pinned to one CPU. - The full `differential_oracle` test binary passes (9/9). - I could not reproduce the CI failure locally even with the server pinned, so the fix comes from tracing the timing in the code, not from a failing local run. The changes in `consumers.rs` and `mod.rs` are not committed yet
Intent
I want you to really hyper focus on making the symbolic engine, you know, crisp and really truly production ready. (Context: #292 : at 1,000 sessions under saturation, subscribers get missing and stray deltas; overload must degrade latency or refuse work, never deliver wrong or lost deltas.)
What Changed
src/protocol/subscription/parameterized.rs, each family runs one round at a time, guarded by a newevaluatinglock. ARoundnow startsPendingand takes its knowledge-graph snapshot only when its turn comes. Views that need a newer revision while a round is evaluating join the single pending round. Under saturation a shape therefore costs one evaluation at a time, and a delta waits for at most two of them. Unit tests inparameterized/tests.rscover the pending and started round states, using a test-onlyround_gate.tests/e2e_reactive/saturation.rs, which runs the load from issue Saturation at 1,000 sessions: deltas arrive seconds late (shared rounds multiply per revision, writes starved) #292. It opens 960 sessions, each subscribed to its own bound standing query, while writers saturate the engine for 30 s. The test asserts that each probe delta reaches exactly its own session, once, withinDELTA_BOUNDof the write's acknowledgement, and that no session gets a stray delta or asubscription_error. The testkitAgentgainspoll_deltaandclear_noticesto support it.websocket-api.mdx,configuration.md,config.example.toml, theSubscriptionsConfigdoc comment, the regenerated GUI docs bundle, and a newTESTING.mdentry for the saturation test. An earlier change on this branch that split compute permits between requests and standing evaluations was reverted, so the net diff leaves them sharing the same permits.Risk Assessment
✅ Low: I found no issues. The fix-round change is small: every probe delta is now checked against DELTA_BOUND inside
settle, and the latency it measures uses the frame's receive time, stamped by the client's dedicated reader task. The production change is the pipelined family rounds, one evaluating and one pending, with the snapshot taken at start under the state lock. I traced it for join, start, cancellation and rule-change races and it still holds its invariants.Testing
I built the real server binary and drove the issue #292 saturation scenario through the e2e_reactive harness. The setup was 960 WebSocket sessions, each subscribed to its own bound standing query, plus 32 writers saturating the engine for 30 s and a prober changing one session at a time. It passed on 32 cores, and with taskset on 4 and 2 cores. Max ack-to-delta latency was 2.8–3.2 s against the 5 s bound, and most deltas were delivered during the load. Running the same test against the base commit's engine source failed with a 12 s late delta, so the test reproduces the regression and the fix resolves it. The rest of the e2e_reactive suite (14 tests) and the targeted parameterized round tests also passed. The worktree is clean; logs are in the evidence directory.
Evidence: Saturation test, 32 cores (pass)
saturated sessions: 960 sessions, 11706 receipts in 30s, 187 probes, 180 delivered during the load; ack->delta p50 806ms p99 1.81s max 2.79sEvidence: Saturation test pinned to 4 cores (pass)
960 sessions, 2971 receipts in 30s, 77 probes, 73 delivered during the load; ack->delta p50 543ms p99 2.24s max 2.89sEvidence: Saturation test pinned to 2 cores (pass)
960 sessions, 1449 receipts in 30s, 41 probes, 38 delivered during the load; ack->delta p50 1.04s p99 1.70s max 3.19sEvidence: Saturation test against base engine source (fails, reproduces #292)
Error: Timeout("probe delta of session s-85 arrived 12.028645344s after its ack, beyond 5s")Evidence: Remaining e2e_reactive tests
Evidence: Parameterized round unit/integration tests
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 1 issue found → auto-fixed ✅
gui/lib/docs-bundle.ts- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Review** - 1 issue found → auto-fixed ✅
tests/e2e_reactive/saturation.rs:220- The module doc (lines 9-11) says each probe's delta arrives "within DELTA_BOUND of the write's acknowledgement, so while the writers run". Only probes still outstanding when the load ends are held to that bound, by thetimeout_at(oldest + DELTA_BOUND)loop at line 259. A delta that arrives during the 30 s load goes throughsettle(line 152), which records its latency but never compares it with DELTA_BOUND. Example: a probe acknowledged at t=2 s whose delta arrives at t=25 s is accepted, and its 23 s only shows up in the eprintln max. So the required pass does not check the delivery-time bound that the issue Saturation at 1,000 sessions: deltas arrive seconds late (shared rounds multiply per revision, writes starved) #292 regression is meant to catch while saturated. Fix: insettle(or at the line 220 call site), fail withViolation::Timeoutwhenafter_ack > DELTA_BOUND. That covers the during-load path and the line 261 path together.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
cargo test --release --test e2e_reactive -- saturation --nocapture(32 cores, real inputlayer-server binary)taskset -c 0-3 cargo test --release --test e2e_reactive -- saturation --nocapturetaskset -c 0-1 cargo test --release --test e2e_reactive -- saturation --nocapturegit checkout 57ad2e95 -- src && cargo test --release --test e2e_reactive -- saturation --nocapture(base engine with the new test; src restored to HEAD afterwards)cargo test --lib subscription::parameterizedcargo test --test standing_query_tests parameterizedcargo test --release --test e2e_reactive -- --skip saturation✅ **Document** - passed
✅ No issues found.
src/ast/mod.rs:709-make doc-check(RUSTDOCFLAGS=-D warnings cargo doc) fails on a broken intra-doc link:crate::params::bind_statementsdoes not exist. This change does not touch the file; the break came earlier. Follow-up: point the link at the current binding function, or turn it into plain code text.✅ **Push** - passed
✅ No issues found.