Repository navigation
fix: refuse or stop runaway cross-product queries within their limits and release their memory - #304
Merged
Merged
Conversation
jsam
changed the title
…away One cartesian query (?a(X), b(Y) over 30,000 rows each) on shipped defaults still drove the server to its memory ceiling and was OOM-killed, even with the per-query memory limit: differential dataflow's join emits every pair a key matches in one step, a cross product's pairs all share one key, and the deadline, cancel and memory limit were checked only between steps. - Joins, result sinks and the antijoin's negated side now check the request every 1,024 pairs or rows they form, inside the step: a cross product is stopped a few KB past its memory limit (resource_exhausted) or within milliseconds of its deadline or cancel. The antijoin sub-dataflow checks between its steps too, and a failure there now fails the query instead of treating the negated side as empty. - storage.performance.max_query_cost now estimates rows from relation sizes (a join on shared variables as its larger input, a cross product as the product of its inputs) and defaults to 100,000,000: an obvious cross product is refused with validation before it runs. A startup warning names max_query_cost = 0. - storage.performance.max_recursion_iterations (default 100,000) stops recursion that keeps deriving new values, whatever the deadline. - A graph loaded at startup served its queries before its first write without the configured max_query_cost and max_result_rows; its first snapshot now carries the limits. - A stopped computation keeps its compute permit until its thread exits, as before; the server now logs when it exits (or that it is still running after a second), and a computation that held over 64 MiB returns the memory it freed to the operating system, so RSS falls back to baseline. - The default budget of all queries together now follows the tightest cgroup limit on the process, so a systemd unit's MemoryMax= sets it (it read only the cgroup root, and found none). A query holding under 1/64 of that budget (at most 16 MiB) is no longer stopped when others fill it: the queries that filled it are. Measured on a 6 GB-capped release server: the 30k x 30k cross product is refused in 1 ms; with max_query_cost = 0 it stops at 4 GiB held (25 KB past the limit) and RSS returns from 4.8 GB to 53 MB; six at once are each stopped by the server budget (peak RSS 4.3 GB) while a tenant on another graph keeps answering in 1-8 ms with no errors; a client disconnect stops the computation and frees its memory.
…ling off by default
…s in this PR's code. **What failed:** the differential oracle check `examples/iql/09_recursion/31_mutual_binary_paths.iql`. In the `recompute` and `subscription` adapters, `?even_path(X, 4)` returned no rows instead of (1,4), (2,4), (3,4). It reproduced locally and passed once the new Distinct pushdown was removed. **Root cause:** this PR added a filter pushdown through `Distinct` in `Optimizer::pushdown_filters`, so filters can now reach the existing Join pushdown code more often. That code moved a right-side predicate by subtracting the number of left columns. It assumed a join row is all left columns followed by all right columns. In fact the engine builds a join row as the left row followed by the right row without its key columns (`left_tuple.concat(right_tuple.excluding_indices(right_keys))` in the code generator, and `build_join` builds the output schema the same way). So in `Join(odd_path(X,Y), edge(Y,Z))` with output [X,Y,Z], the filter `Z = 4` became `edge.col0 (Y) = 4`. **Rule that must hold:** a filter moved below a join must test the same column it tested on the join's output. The left side was already correct, because a join row starts with the full left row. The right side is the only other place this applies, and it is fixed. **Fix (`src/optimizer/mod.rs`):** - A new helper, `right_input_predicate`, maps each join output column to its right-input column, skipping the key columns. It uses the existing `Predicate::adjust_for_projection`, which handles every predicate variant. If a predicate can't be mapped, it stays above the join. - `adjust_predicate_columns` and its six unit tests are removed, since nothing calls it any more. - `test_pushdown_filter_to_right` used a join layout the engine never builds (the key column listed twice). It now uses the real [x,y,z] layout and checks that `z < 100` lands on `s` column 1. The old code would have put it on column 0, so this test would have failed before the fix. **Verified:** the differential oracle test passes (5/5), the optimizer unit tests pass (52), `make unit-test` (`cargo test --workspace --all-features`) exits 0, and `cargo clippy --workspace --all-targets --all-features -- -D warnings` is clean
jsam
force-pushed
the
fm/fix-runaway-query
branch
from
October 5, 2026 12:44
36f0f15 to
803cc6e
Compare
This was referenced Oct 5, 2026
Merged
jsam
added a commit
that referenced
this pull request
Oct 5, 2026
…saves most of the work (#378) (#385) * fix(protocol): share keyed subscription families that save most of the work, and judge again as views join (#378) A family of keyed subscriptions (one bound view per session, `?r1(H_k, Y)`) shared one evaluation per write only while a round was no slower than the views' own evaluations run in parallel on the compute permits. That compares latency on an idle server and ignores the work: with 18 permits a family accepted up to 18 times the CPU for equal latency, and every own evaluation holds a permit that writes and reads wait for. #304 made bound rule reads about five times cheaper (`?r1(400,Y)` at 100k edges: 6.3 to 1.2 ms). Own evaluations then looked faster in parallel than a round, so keyed families stopped sharing (logged on main at 1M edges: `shared_us=85445 own_us=9739 bindings=100 permits=18`), and the backoff after that verdict outlasted the 1,000 views that joined later. On the views benchmark a write with 100 keys at 100k edges cost 154 ms of server CPU instead of 46, and with 1,000 keys at 1M edges 11.9 s and a 643 ms ack instead of 0.18 s and 3.8 ms. - A round now also keeps sharing when it does at most half the views' own evaluations' combined work. - A family that stopped sharing probes again as soon as it has twice the bindings it had when it stopped, instead of waiting out a backoff that grows with the bindings: a verdict reached while the first views subscribed no longer holds for the many that follow. On the benchmark host (servers on 24 CPUs, 18 compute permits) the issue's case, 1M edges with keyed 1, 10, 100 then 1,000 subscribers, now costs 117 ms of server CPU per write at 1,000 keys, with a 4.0 / 4.3 ms ack p50/p99 and a 75 / 79 ms delta p50/p99 (main: 11,812 ms, 649 / 687 ms, 689 / 1,212 ms; before #304: 180 ms, 3.8 / 8.3 ms, 75 / 104 ms). At 100 keys on 100k edges: 17 ms per write (main 154, before #304 46). * test(perf-gate): gate keyed subscriptions on delta latency and server CPU per write (#378) `delta_keyed`: 300 agents, each subscribed to its own key `?r1(H_k, Y)` of a deployed rule over 100K edges; an open-loop writer adds one row to one key every 20 ms. Every write refreshes all 300 views, which differ only in their constant, so the engine must evaluate the family once per write rather than each view's own query. Besides delta and ack latency, the fixture reports `writes_per_server_cpu_sec`, writes per second of server CPU over the write phase, which shows lost sharing even on a host idle enough for the deltas to arrive in time. Its delta p50/p99, ack p99 and CPU rate join the policy's required set, and the fixture joins the gate's default fixtures. Six rounds on the benchmark host. Against the commit before #304 (6ac5322), every required metric passes: delta p50 10.5 ms on both, delta p99 29.7 to 16.6 ms, ack p99 20.3 to 9.6 ms, 2.3 times less server CPU per write. Against current main the fixture shows the trade the sharing fix makes, which is accepted: 6 to 8 times less server CPU per write and a better tail under load (at a write every 10 ms main's delta p99 reaches 51 ms, the fix's stays under 19 ms), for a higher idle delta p50 (5.8 to 10.2 ms), because main evaluates all 300 views in parallel at that CPU cost. * no-mistakes(review): Cap keyed sharing at twice the parallel own latency * no-mistakes(document): Clarify share_parameterized config comment; rustfmt keeps_sharing
jsam
added a commit
that referenced
this pull request
Oct 6, 2026
…, not after every one (#384) Recursive reads were 9-20% slower since #304. The bisect blamed the per-step deadline and memory checks inside recursion, but on the benchmark host at main they cost nothing (?hot_reach(X,Y) over 100K edges: 275.7 ms with them, 275.1 ms with them stubbed). The cost is the release of freed memory #304 added: after every computation that peaked over 64 MiB, the server purged the allocator's freed pages on the reply path, and the next read faulted the same pages back in. With that release stubbed the same read takes 223.5 ms (-19%), ?path(400,Y) 220.7 ms instead of 265.4 ms. The release now waits until no large computation has finished for one second, then runs once on the blocking pool: a stream of large reads keeps reusing its pages, and the memory goes back to the operating system within a second of the last one. The stop guards are untouched, and the memory a stream holds meanwhile is bounded as before by the limits its computations ran under; resident memory can stay at the stream's working size for up to a second longer than before. The perf gate now has the case: `recursive_query`, `?hot_reach(X, Y)` of the views benchmark's rules over the delta_keyed graph (100K edges), 25 serial reads per round, judged on p50.
jsam
added a commit
that referenced
this pull request
Oct 6, 2026
…, not after every one (#384) Recursive reads were 9-20% slower since #304. The bisect blamed the per-step deadline and memory checks inside recursion, but on the benchmark host at main they cost nothing (?hot_reach(X,Y) over 100K edges: 275.7 ms with them, 275.1 ms with them stubbed). The cost is the release of freed memory #304 added: after every computation that peaked over 64 MiB, the server purged the allocator's freed pages on the reply path, and the next read faulted the same pages back in. With that release stubbed the same read takes 223.5 ms (-19%), ?path(400,Y) 220.7 ms instead of 265.4 ms. The release now waits until no large computation has finished for one second, then runs once on the blocking pool: a stream of large reads keeps reusing its pages, and the memory goes back to the operating system within a second of the last one. The stop guards are untouched, and the memory a stream holds meanwhile is bounded as before by the limits its computations ran under; resident memory can stay at the stream's working size for up to a second longer than before. The perf gate now has the case: `recursive_query`, `?hot_reach(X, Y)` of the views benchmark's rules over the delta_keyed graph (100K edges), 25 serial reads per round, judged on p50.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
I want you to really hyper focus on making the symbolic engine, you know, crisp and really truly production ready. (Context: #296 and audit report section P0-5: one cartesian query on shipped defaults exhausted memory and was OOM-killed; the 30 s timeout never fired.) The P0-5 finding being fixed here: a single authenticated query such as ?a(X), b(Y) over two relations of 30,000 rows each, on shipped default configuration, drove the server process to its memory ceiling and got it OOM-killed; the 30-second query timeout never fired, and the client disconnecting did not free the memory either. Scope of this change: that P0 runaway query must be refused or cancelled within the configured limits with a typed error, and its memory released, so the server stays alive and other tenants keep answering.
What Changed
src/code_generator). A runaway query now stops withresource_exhaustedright after it passes its memory limit, or soon after its deadline or a cancel. If the antijoin sub-dataflow fails, the query now fails too; before, the negated side was treated as empty. After a stopped computation that held more than the release threshold, the server returns the freed memory to the OS before handing the compute permit to the next query. It also logs when the computation exits, or that it is still running after 1 s.storage.performance.max_query_costnow limits the estimated rows per join, based on relation sizes. A join on shared variables counts as its larger input; a cross product counts as the product of its inputs. Plans over the limit are estimated again using only the rows their filters keep of stored relations. The default changes from 0 to 100,000,000, so?a(X), b(Y)over 30k × 30k is refused withvalidationbefore it runs, and the server warns at startup when the limit is set to 0. Also added:max_recursion_iterationssetting (default 0 = off) that stops recursion that never reaches a fixpoint.Distinct, and a filter moved to a join's right side now maps to the correct column, skipping the join keys.MemoryMax=), not just the cgroup root. A query holding less than 1/64 of the budget (at most 16 MiB) is no longer stopped when other queries fill the pool; the queries that filled it are. The config example, configuration/deployment guides and the error spec (with new "Query Too Complex" and "Recursion Did Not Reach a Fixpoint" entries) are updated, andtests/memory_limit_tests.rsadds coverage.Risk Assessment
Testing
I drove the real release server over WebSocket, plus the bundled CLI client, inside an 8 GiB memory-capped scope standing in for a container. On shipped defaults, the P0-5 cross product is refused right away with
validation. With the cost guard off, it is stopped by the memory limits, the deadline, a cancel or a disconnect, each with a typed error. Memory came back to idle every time, other tenants kept answering, and there was no OOM kill. The operator's decisions on filtered queries and the off-by-default recursion ceiling hold, and the mutual-recursion fix gives the expected rows. Every scenario passed. The artifacts are CLI/WebSocket transcripts and cgroup counters in the evidence directory; there is no UI surface.Evidence: Shipped-defaults transcript: P0-5 refused with validation, other tenant answers, filtered queries run, wide filters refused
Evidence: Cross product through persistent and session rules refused on defaults
Evidence: Runaway with guard off: stopped by memory limit, total memory limit, deadline, cancel and disconnect, with RSS samples
Evidence: Scope cgroup counters: 8 GiB max, 5.8 GB peak, oom_kill 0
Evidence: Unbounded recursion on defaults: deadline_exceeded, no iteration refusal
Evidence: Mutual-recursion example run through inputlayer-client (matches .iql.out)
Evidence: Server logs (defaults and guard-off runs)
Evidence: Key runaway outcomes
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ No issues found.
✅ No issues found.
🔧 **Test** - 1 issue found ✅
docs/content/docs/guides/websocket-api.mdx:140- The docs say a stopped query 'stops computing within milliseconds'. In a live run with both memory limits off (max_query_memory_bytes=0, max_total_query_memory_bytes=0), the deadline reply for ?a(X), b(Y) arrived at 30.00s, but the computation loggedquery_stopped_still_running waited_ms=1001and only exited after 5007 ms, while freeing 19 GB. The error itself is prompt and the memory is released, so this does not violate the intent. With the default memory limits, nothing of this size builds up. The wording overstates exit latency only when the limits are disabled.validation, and the server stays aliveresource_exhausted, there is no OOM kill, memory is released, and another tenant answers throughoutcargo build --release --bin inputlayer-server(isolated CARGO_TARGET_DIR under /tmp)Live: shipped-defaults server (no config file) in asystemd-run --user --scope -p MemoryMax=6Gscope, driven over /ws by a throwaway Python websockets client: ?a(X), b(Y); ?a(X), b(Y), !c(Y); other-KG ?t(X); ?a(X), b(X), X < 3; ?a(7), b(Y); ?a(X), X > 29998, b(Y); ?a(X), X < 20000, b(Y); ?a(X), X > 5, b(Y), Y > 5; session rule cross(X,Y) <- a(X), b(Y) then ?cross(X,Y); converging recursion upto(N) to N < 3000Live: server with INPUTLAYER_STORAGE__PERFORMANCE__MAX_QUERY_COST=0 in 6G cgroup: ?a(X), b(Y) while a second tenant queried every second, RSS sampled, cgroup memory.events checkedLive: cost guard and both memory limits off, 6G cgroup: client closes the socket 1 s into ?a(X), b(Y), RSS sampled for 5 s, other tenant queriedLive: cost guard and both memory limits off, 60G cgroup: ?a(X), b(Y) under the default 30 s deadline, RSS sampled until memory released, supervise logs capturedLive: defaults server, recursion running 100,500 iterations (upto(N) ... N < 100500)cargo test --test memory_limit_tests(12 end-to-end /ws tests, including the new cross-product, filter-aware estimate, memory-stop, deadline-stop and iteration-ceiling tests)✅ No issues found.
cargo build --release --bin inputlayer-server --bin inputlayer-clientRanstart-server.sh defaults: inputlayer-server with shipped defaults in asystemd-run --user --scope -p MemoryMax=8Gscope. The startup log shows total_query_memory_bytes detected from the cgroup as 60% of 8 GiB.node load.mjs: loaded a, b and a2 with 30,000 rows each over WebSocketnode s_defaults.mjs: P0-5 cartesian query, a second tenant querying at the same time, filtered and comparison cross products (one match vs. 20,000 kept), and a keyed joinnode s_rules.mjs: cartesian query through a persistent rule and through a session rulenode s_recursion.mjson defaults: unbounded counter with timeout_ms 5000 while another tenant queries, then the default 30 s deadlineinputlayer-client ... examples/iql/09_recursion/31_mutual_binary_paths.iql, output diffed against the committed .iql.outRanstart-server.sh guardoff guard-off.toml(max_query_cost=0, max_recursion_iterations=1000) in the same 8 GiB scopenode s_runaway.mjs: the query stopped by the per-query memory limit, by the total memory limit with two tenants, by timeout_ms 300, by an explicit cancel, and by a client disconnect. Server RSS was sampled throughout, and the recursion ceiling was checked with it set explicitly.Read the cgroupmemory.peakandmemory.events(oom/oom_kill), and searched the server log for panics and query_stopped_exited✅ **Document** - passed
✅ No issues found.
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
✅ No issues found.