Skip to content

feat(execution): refuse queries and writes over memory limits with resource_exhausted - #275

Merged
jsam merged 6 commits into
mainfrom
fm/en-en9
Oct 4, 2026
Merged

feat(execution): refuse queries and writes over memory limits with resource_exhausted#275
jsam merged 6 commits into
mainfrom
fm/en-en9

Conversation

@jsam

@jsam jsam commented Oct 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Intent

please continue developing the input layer engine so we make sure that the symbolic engine is ready to go live and completely production ready while we develop the GoTM uh, market.

(Context: the captain approved the neurosymbolic decision engine plan; its milestone 'Engine readiness' makes the symbolic engine production-ready for the hosted decision API, where each tenant environment runs on an engine cell. This task is item EN-9 of that milestone: per-graph memory budget and per-query memory limit: refuse with resource_exhausted instead of growing (new; today an inputlayer-server reached about 115 GiB running a runaway query and took the host down) — why: one tenant's query must never take down its cell, and the process must stay inside its container limit so it is not killed)

What Changed

  • Adds a metered global allocator (src/execution/memory.rs) that counts net heap bytes per thread. Request control charges each query's growth at the evaluator's cooperative checkpoints and stops a query once it goes over max_query_memory_bytes (default 4 GiB), or once it grows while all in-flight queries together are over max_total_query_memory_bytes. That total defaults to 60% of the cgroup v2/v1 memory limit, or no limit outside a container. A query stopped this way applies nothing, and the limit is also enforced after commits.
  • Adds a per-graph max_graph_memory_bytes budget based on an estimate of each graph's stored facts. Program and fact commits that would grow a graph past it are refused with StorageError::MemoryBudgetExceeded. Deletes are always accepted, and recovery can still load a graph that is over budget.
  • Adds a resource_exhausted error code to the WS protocol and maps every memory stop to it. The code is also added to the JS/Python SDK protocol types, the GUI types, the AsyncAPI spec and the error docs. The branch also documents the new settings and a sizing rule in the configuration and deployment guides, config.example.toml and the Kubernetes configmap, and raises the StatefulSet memory limit from 4Gi to 8Gi. A new tests/memory_limit_tests.rs suite covers the limits.

Risk Assessment

⚠️ Medium: No defect could be substantiated with a concrete sequence. The prior fix rounds hold up in the current code: queries charge one shared count per query and the global pool by delta at checkpoints. Memory stops after a commit are reported as resource_exhausted on the query, .why and fact-staging paths, and tests cover those sequences. Every request path, including subscriptions and REST, runs under the limited control. The graph-budget byte count is kept in step by every RelationStore mutation. The docs, config and k8s sizing values match the user's 0.8× rule. Risk stays medium because the change installs a process-wide global allocator and adds new stop paths to every query evaluation.

Testing

I built the release server and drove it over WebSocket with a Node client. Each scenario ran on its own fresh server in a temp data dir. A memory-limited Docker container with its own cgroup namespace covered the container-limit case. All six live scenarios passed: - A runaway closure is stopped in about 300 ms with resource_exhausted. The message says nothing was applied, and the connection keeps serving. - After a commit, the query, .why and conditional-delete paths all report resource_exhausted, not validation. The delete applies nothing. - A write past the graph budget is refused atomically. Deletes and re-inserts still work, and each graph has its own budget. - Four concurrent runaways are each refused by the global pool, and the server keeps answering. - In a 1 GiB container with the shipped defaults, the derived 614 MiB pool (60% of 1 GiB) refused the runaway at a container peak of 632 MiB, with no OOM kill. - With the limits disabled, the same container was OOM-killed (exit 137), which reproduces the original incident. The exact under-limit 6-row case was covered only by an in-process integration test, so it is marked untested here. The 6 focused integration tests and the cgroup-default unit test also pass. There is no UI surface, so the evidence is CLI and WebSocket transcripts. No temp dirs, containers or server processes were left behind, and the worktree is clean.

  • Live validation: ✅ go - 6 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Tenant runs an unbound transitive closure over a 1500-node graph with max_query_memory_bytes=32MiB: gets resource_exhausted naming the limit and 'nothing was applied'; the same connection then answers… ✅ pass live s1-runaway.txt (refused in 299 ms, server RSS 78 MiB, follow-up queries return results)
Runaway computation after the program began committing (.compact + closure query, .compact + .why, rule drop + conditional delete over closure) is reported as resource_exhausted, not validation, and t… ✅ pass live s2-after-commit.txt (all three error code resource_exhausted; edge rows 4496 before and after)
Write that would grow a graph past max_graph_memory_bytes=64KiB is refused with resource_exhausted and applies nothing, including earlier statements in the same program; deletes are accepted and free… ✅ pass live s3-graph-budget.txt
Adversarial: four concurrent runaway closures, each under its 1 GiB per-query limit, are all refused once together they exceed max_total_query_memory_bytes=64MiB; the server still serves afterwards ✅ pass live s4-global-pool.txt (4x resource_exhausted naming max_total_query_memory_bytes, ServerMemoryExhausted in log)
Process stays inside its container: in a 1 GiB Docker container with shipped defaults (no explicit pool), the pool derives to 60% of the cgroup limit, the runaway is refused and the container is not O… ✅ pass live s5b-container-shipped-defaults.txt (pool_budget_bytes=644245092 = 0.6 x 1GiB, memory.peak 632 MiB, oom_kill 0, running=true)
Baseline contrast: same container and query with both limits disabled is OOM-killed, reproducing the original incident the guard prevents ✅ pass live s5a-container-limits-disabled.txt (OOMKilled=true exit=137, startup warns max_query_memory_bytes=0)
A query under the limit is unaffected (closure over a tiny graph returns all 6 rows) ⏸️ untested no The prior payload did not establish a live result for this scenario. The exact 6-row case was run only as the in-process integration test a_query_under_the_limit_is_unaffected, not against the running…
Evidence: Runaway query refused under per-query limit 
>> [create-runaway] .kg create runaway
<< (5 ms) {"type":"result","id":"create-runaway","columns":["message"],"rows":[["Knowledge graph 'runaway' created."],["Switched to knowledge graph: runaway"]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":4,"row_provenance":["unknown","unknown"],"switched_kg":"runaway","errors":[]}
>> [load] +edge[(0, 1), (0, 0), (0, 1), (1, 2), (1, 2), (1, 4), (2, 3), (2, 4), (2, 7), (3, 4), (3, 6), (3, 10), (4, 5), (4, 8), (...(56348 chars)
<< (15 ms) {"type":"result","id":"load","columns":["message"],"rows":[["Inserted 4496 fact(s) into 'edge'."]],"row_count":1,"total_count":1,"truncated":false,"execution_time_ms":7,"row_provenance":["unknown"],"errors":[],"statements":[{"index":0,"kind":"insert","inserted":4496,"deleted":0}]}
PASS: 1500-node edge graph (4500 facts) loads
server before closure: RSS 38 MiB, peak 38 MiB
>> [closure] reach(X, Y) <- edge(X, Y) \n reach(X, Z) <- reach(X, Y), edge(Y, Z) \n ?reach(X, Y)
<< (299 ms) {"type":"error","id":"closure","message":"Request exceeded the per-query memory limit (storage.performance.max_query_memory_bytes) before it began committing; nothing was applied. Narrow the query or bind more of its arguments","code":"resource_exhausted"}
PASS: unbound transitive closure refused with resource_exhausted
PASS: message names the limit and says nothing was applied
server after closure: RSS 78 MiB, peak 78 MiB
>> [bound] ?edge(0, Y)
<< (1 ms) {"type":"result","id":"bound","columns":["0","Y"],"rows":[[0,0],[0,1]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":0,"row_provenance":["unknown","unknown"],"timing_breakdown":{"total_us":331,"parse_us":22,"sip_us":6,"magic_sets_us":3,"ir_build_us":10,"optimize_us":32,"shared_views_us":0},"errors":[]}
PASS: same connection still answers a small query
>> [small-closure] reach(X, Y) <- edge(X, Y) \n reach(X, Z) <- reach(X, Y), edge(Y, Z) \n ?reach(0, 1)
<< (7 ms) {"type":"result","id":"small-closure","columns":["0","1"],"rows":[[0,1]],"row_count":1,"total_count":1,"truncated":false,"execution_time_ms":6,"row_provenance":["unknown"],"timing_breakdown":{"total_us":6564,"parse_us":20,"sip_us":13,"magic_sets_us":33,"ir_build_us":15,"optimize_us":81,"shared_views_us":0},"errors":[]}
PASS: bounded closure from one node still answers
server process still alive after scenario
�[2m2026-10-04T20:11:01.499805Z�[0m �[33m WARN�[0m �[2minputlayer::execution::request_control�[0m�[2m:�[0m query_memory_limit_exceeded �[3mheld_bytes�[0m�[2m=�[0m50612855 �[3mlimit_bytes�[0m�[2m=�[0m33554432 �[3mpool_held_bytes�[0m�[2m=�[0m50612855 �[3mpool_budget_bytes�[0m�[2m=�[0m0 �[3mreason�[0m�[2m=�[0mSome(MemoryExhausted)
Evidence: Post-commit query / .why / conditional delete report resource_exhausted 
>> [create-compact] .kg create compact
<< (5 ms) {"type":"result","id":"create-compact","columns":["message"],"rows":[["Knowledge graph 'compact' created."],["Switched to knowledge graph: compact"]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":3,"row_provenance":["unknown","unknown"],"switched_kg":"compact","errors":[]}
>> [load] +edge[(0, 1), (0, 0), (0, 1), (1, 2), (1, 2), (1, 4), (2, 3), (2, 4), (2, 7), (3, 4), (3, 6), (3, 10), (4, 5), (4, 8), (...(56348 chars)
<< (16 ms) {"type":"result","id":"load","columns":["message"],"rows":[["Inserted 4496 fact(s) into 'edge'."]],"row_count":1,"total_count":1,"truncated":false,"execution_time_ms":9,"row_provenance":["unknown"],"errors":[],"statements":[{"index":0,"kind":"insert","inserted":4496,"deleted":0}]}
>> [rules] +reach(X, Y) <- edge(X, Y) \n +reach(X, Z) <- reach(X, Y), edge(Y, Z) \n +spare(X) <- edge(X, 0)
<< (8 ms) {"type":"result","id":"rules","columns":["message"],"rows":[["Rule 'reach' registered."],["Rule 'reach' registered."],["Rule 'spare' registered."]],"row_count":3,"total_count":3,"truncated":false,"execution_time_ms":6,"row_provenance":["unknown","unknown","unknown"],"errors":[]}
edge rows before: 4496
>> [compact-query] .compact \n reach(X, Y) <- edge(X, Y) \n reach(X, Z) <- reach(X, Y), edge(Y, Z) \n ?reach(X, Y)
<< (264 ms) {"type":"result","id":"compact-query","columns":["message"],"rows":[["Compaction complete."],["Session rule added for 'reach'."],["Session rule added for 'reach'."],["Query execution failed: Query exceeded the per-query memory limit (storage.performance.max_query_memory_bytes). Narrow the query or bind more of its arguments"]],"row_count":4,"total_count":4,"truncated":false,"execution_time_ms":262,"row_provenance":["unknown","unknown","unknown","unknown"],"errors":[{"index":3,"code":"resource_exhausted","message":"Query execution failed: Query exceeded the per-query memory limit (storage.performance.max_query_memory_bytes). Narrow the query or bind more of its arguments"}]}
PASS: .compact then closure query: query statement refused resource_exhausted
>> [compact-why] .compact \n .why ?reach(X, Y)
<< (317 ms) {"type":"result","id":"compact-why","columns":["message"],"rows":[["Compaction complete."],["Why error: Query execution failed: Query exceeded the per-query memory limit (storage.performance.max_query_memory_bytes). Narrow the query or bind more of its arguments"]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":316,"row_provenance":["unknown","unknown"],"errors":[{"index":1,"code":"resource_exhausted","message":"Why error: Query execution failed: Query exceeded the per-query memory limit (storage.performance.max_query_memory_bytes). Narrow the query or bind more of its arguments"}]}
PASS: .compact then .why proof: refused resource_exhausted (not validation)
>> [cond-delete] .rule drop spare \n -edge(X, Y) <- reach(X, Y)
<< (262 ms) {"type":"result","id":"cond-delete","columns":["message"],"rows":[["Delete failed: Query execution failed: Query exceeded the per-query memory limit (storage.performance.max_query_memory_bytes). Narrow the query or bind more of its arguments (rolled back: none of the program's 2 write statements 0-1 was applied)"]],"row_count":1,"total_count":1,"truncated":false,"execution_time_ms":261,"row_provenance":["unknown"],"errors":[{"index":1,"code":"resource_exhausted","message":"Delete failed: Query execution failed: Query exceeded the per-query memory limit (storage.performance.max_query_memory_bytes). Narrow the query or bind more of its arguments (rolled back: none of the program's 2 write statements 0-1 was applied)"}]}
PASS: rule drop then conditional delete over closure: refused resource_exhausted (not validation)
edge rows after: 4496
PASS: conditional delete applied nothing
server: RSS 82 MiB, peak 82 MiB
server process still alive after scenario
�[2m2026-10-04T20:11:06.917508Z�[0m �[33m WARN�[0m �[2minputlayer::execution::request_control�[0m�[2m:�[0m query_memory_limit_exceeded �[3mheld_bytes�[0m�[2m=�[0m34373220 �[3mlimit_bytes�[0m�[2m=�[0m33554432 �[3mpool_held_bytes�[0m�[2m=�[0m34373220 �[3mpool_budget_bytes�[0m�[2m=�[0m0 �[3mreason�[0m�[2m=�[0mSome(MemoryExhausted)
�[2m2026-10-04T20:11:07.233761Z�[0m �[33m WARN�[0m �[2minputlayer::execution::request_control�[0m�[2m:�[0m query_memory_limit_exceeded �[3mheld_bytes�[0m�[2m=�[0m50614372 �[3mlimit_bytes�[0m�[2m=�[0m33554432 �[3mpool_held_bytes�[0m�[2m=�[0m50614372 �[3mpool_budget_bytes�[0m�[2m=�[0m0 �[3mreason�[0m�[2m=�[0mSome(MemoryExhausted)
�[2m2026-10-04T20:11:07.495526Z�[0m �[33m WARN�[0m �[2minputlayer::execution::request_control�[0m�[2m:�[0m query_memory_limit_exceeded �[3mheld_bytes�[0m�[2m=�[0m50626922 �[3mlimit_bytes�[0m�[2m=�[0m33554432 �[3mpool_held_bytes�[0m�[2m=�[0m50626922 �[3mpool_budget_bytes�[0m�[2m=�[0m0 �[3mreason�[0m�[2m=�[0mSome(MemoryExhausted)
Evidence: Per-graph budget refuses growth, deletes still pass 
>> [create-budget] .kg create budget
<< (4 ms) {"type":"result","id":"create-budget","columns":["message"],"rows":[["Knowledge graph 'budget' created."],["Switched to knowledge graph: budget"]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":3,"row_provenance":["unknown","unknown"],"switched_kg":"budget","errors":[]}
>> [small] +edge[(0, 1), (0, 0), (0, 1), (1, 2), (1, 2), (1, 4), (2, 3), (2, 4), (2, 7), (3, 4), (3, 6), (3, 10), (4, 5), (4, 8), (...(3076 chars)
<< (5 ms) {"type":"result","id":"small","columns":["message"],"rows":[["Inserted 298 fact(s) into 'edge'."]],"row_count":1,"total_count":1,"truncated":false,"execution_time_ms":3,"row_provenance":["unknown"],"errors":[],"statements":[{"index":0,"kind":"insert","inserted":298,"deleted":0}]}
PASS: small write within 64 KiB graph budget accepted
edge rows: 298
>> [big] +edge[(100, 101), (100, 200), (100, 301), (101, 102), (101, 202), (101, 304), (102, 103), (102, 204), (102, 307), (103, ...(14572 chars)
<< (2 ms) {"type":"error","id":"big","message":"Knowledge graph 'budget' would hold about 155792 bytes of facts, over its memory budget of 65536 bytes (storage.performance.max_graph_memory_bytes); nothing was applied. Delete facts or raise the budget.","code":"resource_exhausted"}
PASS: write past graph budget refused with resource_exhausted naming max_graph_memory_bytes
PASS: nothing applied (rows 298)
>> [program] +note(1)
+edge[(100, 101), (100, 200), (100, 301), (101, 102), (101, 202), (101, 304), (102, 103), (102, 204), (102, 307...(14581 chars)
<< (2 ms) {"type":"result","id":"program","columns":["message"],"rows":[["Knowledge graph 'budget' would hold about 155872 bytes of facts, over its memory budget of 65536 bytes (storage.performance.max_graph_memory_bytes); nothing was applied. Delete facts or raise the budget. (rolled back: none of the program's 2 write statements 0-1 was applied)"]],"row_count":1,"total_count":1,"truncated":false,"execution_time_ms":0,"row_provenance":["unknown"],"errors":[{"index":1,"code":"resource_exhausted","message":"Knowledge graph 'budget' would hold about 155872 bytes of facts, over its memory budget of 65536 bytes (storage.performance.max_graph_memory_bytes); nothing was applied. Delete facts or raise the budget. (rolled back: none of the program's 2 write statements 0-1 was applied)"}]}
PASS: program: inserting statement refused
>> [note] ?note(X)
<< (0 ms) {"type":"result","id":"note","columns":[],"rows":[],"row_count":0,"total_count":0,"truncated":false,"execution_time_ms":0,"timing_breakdown":{"total_us":141,"parse_us":13,"sip_us":6,"magic_sets_us":2,"ir_build_us":4,"optimize_us":5,"shared_views_us":0},"errors":[]}
PASS: earlier +note in the same program not applied
>> [delete] -edge(X, Y) <- edge(X, Y)
<< (3 ms) {"type":"result","id":"delete","columns":["message"],"rows":[["Conditional delete: 298 fact(s) deleted from 'edge'."]],"row_count":1,"total_count":1,"truncated":false,"execution_time_ms":2,"row_provenance":["unknown"],"errors":[],"statements":[{"index":0,"kind":"delete","inserted":0,"deleted":298}]}
PASS: delete accepted while near budget
>> [again] +edge[(100, 101), (100, 200), (100, 301), (101, 102), (101, 202), (101, 304), (102, 103), (102, 204), (102, 307), (103, ...(3605 chars)
<< (2 ms) {"type":"result","id":"again","columns":["message"],"rows":[["Inserted 300 fact(s) into 'edge'."]],"row_count":1,"total_count":1,"truncated":false,"execution_time_ms":1,"row_provenance":["unknown"],"errors":[],"statements":[{"index":0,"kind":"insert","inserted":300,"deleted":0}]}
PASS: freed space accepts new facts
>> [create-budget2] .kg create budget2
<< (4 ms) {"type":"result","id":"create-budget2","columns":["message"],"rows":[["Knowledge graph 'budget2' created."],["Switched to knowledge graph: budget2"]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":2,"row_provenance":["unknown","unknown"],"switched_kg":"budget2","errors":[]}
>> [other] +edge[(0, 1), (0, 0), (0, 1), (1, 2), (1, 2), (1, 4), (2, 3), (2, 4), (2, 7), (3, 4), (3, 6), (3, 10), (4, 5), (4, 8), (...(3076 chars)
<< (5 ms) {"type":"result","id":"other","columns":["message"],"rows":[["Inserted 298 fact(s) into 'edge'."]],"row_count":1,"total_count":1,"truncated":false,"execution_time_ms":3,"row_provenance":["unknown"],"errors":[],"statements":[{"index":0,"kind":"insert","inserted":298,"deleted":0}]}
PASS: a second graph has its own budget
server process still alive after scenario
�[2m2026-10-04T20:11:07.981378Z�[0m �[33m WARN�[0m �[2minputlayer::storage_engine::program_commit�[0m�[2m:�[0m graph_memory_budget_exceeded �[3mkg�[0m�[2m=�[0mbudget �[3mprojected�[0m�[2m=�[0m155792 �[3mbudget�[0m�[2m=�[0m65536
�[2m2026-10-04T20:11:07.985530Z�[0m �[33m WARN�[0m �[2minputlayer::storage_engine::program_commit�[0m�[2m:�[0m graph_memory_budget_exceeded �[3mkg�[0m�[2m=�[0mbudget �[3mprojected�[0m�[2m=�[0m155872 �[3mbudget�[0m�[2m=�[0m65536
Evidence: Concurrent runaways refused by global query pool 
>> [create-pool] .kg create pool
<< (5 ms) {"type":"result","id":"create-pool","columns":["message"],"rows":[["Knowledge graph 'pool' created."],["Switched to knowledge graph: pool"]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":3,"row_provenance":["unknown","unknown"],"switched_kg":"pool","errors":[]}
>> [closure-1] reach(X, Y) <- edge(X, Y) \n reach(X, Z) <- reach(X, Y), edge(Y, Z) \n ?reach(X, Y)
<< (95 ms) {"type":"error","id":"closure-1","message":"Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later","code":"resource_exhausted"}
>> [closure-2] reach(X, Y) <- edge(X, Y) \n reach(X, Z) <- reach(X, Y), edge(Y, Z) \n ?reach(X, Y)
<< (102 ms) {"type":"error","id":"closure-2","message":"Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later","code":"resource_exhausted"}
>> [closure-3] reach(X, Y) <- edge(X, Y) \n reach(X, Z) <- reach(X, Y), edge(Y, Z) \n ?reach(X, Y)
<< (103 ms) {"type":"error","id":"closure-3","message":"Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later","code":"resource_exhausted"}
>> [closure-0] reach(X, Y) <- edge(X, Y) \n reach(X, Z) <- reach(X, Y), edge(Y, Z) \n ?reach(X, Y)
<< (399 ms) {"type":"error","id":"closure-0","message":"Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later","code":"resource_exhausted"}
4 concurrent closures finished in 399 ms
PASS: every concurrent runaway closure refused with resource_exhausted
messages: Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later | Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later | Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later | Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later
server: RSS 162 MiB, peak 162 MiB
>> [bound] ?edge(0, Y)
<< (1 ms) {"type":"result","id":"bound","columns":["0","Y"],"rows":[[0,0],[0,1]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":0,"row_provenance":["unknown","unknown"],"timing_breakdown":{"total_us":438,"parse_us":28,"sip_us":7,"magic_sets_us":3,"ir_build_us":11,"optimize_us":50,"shared_views_us":0},"errors":[]}
PASS: server still answers after concurrent runaways
server process still alive after scenario
�[2m2026-10-04T20:11:14.009130Z�[0m �[33m WARN�[0m �[2minputlayer::execution::request_control�[0m�[2m:�[0m query_memory_limit_exceeded �[3mheld_bytes�[0m�[2m=�[0m19900839 �[3mlimit_bytes�[0m�[2m=�[0m1073741824 �[3mpool_held_bytes�[0m�[2m=�[0m67367218 �[3mpool_budget_bytes�[0m�[2m=�[0m67108864 �[3mreason�[0m�[2m=�[0mSome(ServerMemoryExhausted)
�[2m2026-10-04T20:11:14.015511Z�[0m �[33m WARN�[0m �[2minputlayer::execution::request_control�[0m�[2m:�[0m query_memory_limit_exceeded �[3mheld_bytes�[0m�[2m=�[0m21872551 �[3mlimit_bytes�[0m�[2m=�[0m1073741824 �[3mpool_held_bytes�[0m�[2m=�[0m67123429 �[3mpool_budget_bytes�[0m�[2m=�[0m67108864 �[3mreason�[0m�[2m=�[0mSome(ServerMemoryExhausted)
�[2m2026-10-04T20:11:14.015518Z�[0m �[33m WARN�[0m �[2minputlayer::execution::request_control�[0m�[2m:�[0m query_memory_limit_exceeded �[3mheld_bytes�[0m�[2m=�[0m22507431 �[3mlimit_bytes�[0m�[2m=�[0m1073741824 �[3mpool_held_bytes�[0m�[2m=�[0m67143909 �[3mpool_budget_bytes�[0m�[2m=�[0m67108864 �[3mreason�[0m�[2m=�[0mSome(ServerMemoryExhausted)
Evidence: 1 GiB container, limits disabled: OOMKilled (baseline) 
container memory.max: 1073741824
>> [create-info] .kg create info
<< (9 ms) {"type":"result","id":"create-info","columns":["message"],"rows":[["Knowledge graph 'info' created."],["Switched to knowledge graph: info"]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":8,"row_provenance":["unknown","unknown"],"switched_kg":"info","errors":[]}
server before: n/a
container memory.peak: 
container running=false OOMKilled=true exit=137
WARNING: max_query_memory_bytes = 0 (unlimited). One runaway query can take the whole server's memory.
Evidence: 1 GiB container, shipped defaults: refused, peak 632 MiB, no OOM

container memory.max: 1073741824 << (5635 ms) {"type":"error","id":"closure","message":"Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later","code":"resource_exhausted"} container memory.peak: 662818816 oom_kill 0 container running=true OOMKilled=false WARN query_memory_limit_exceeded held_bytes=644260922 limit_bytes=4294967296 pool_budget_bytes=644245092 reason=ServerMemoryExhausted

container memory.max: 1073741824
>> [create-info] .kg create info
<< (6 ms) {"type":"result","id":"create-info","columns":["message"],"rows":[["Knowledge graph 'info' created."],["Switched to knowledge graph: info"]],"row_count":2,"total_count":2,"truncated":false,"execution_time_ms":4,"row_provenance":["unknown","unknown"],"switched_kg":"info","errors":[]}
server before: n/a
>> [closure] reach(X, Y) <- edge(X, Y) \n reach(X, Z) <- reach(X, Y), edge(Y, Z) \n ?reach(X, Y)
<< (5635 ms) {"type":"error","id":"closure","message":"Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later","code":"resource_exhausted"}
PASS: closure refused under container-derived pool
message: Request stopped: the queries running on the server hold their whole memory budget (storage.performance.max_total_query_memory_bytes); it was stopped before it began committing and nothing was applied. Retry later
container memory.peak: 662818816
oom_kill 0
container running=true OOMKilled=false exit=0
�[2m2026-10-04T20:12:04.473731Z�[0m �[33m WARN�[0m �[2minputlayer::execution::request_control�[0m�[2m:�[0m query_memory_limit_exceeded �[3mheld_bytes�[0m�[2m=�[0m644260922 �[3mlimit_bytes�[0m�[2m=�[0m4294967296 �[3mpool_held_bytes�[0m�[2m=�[0m644260922 �[3mpool_budget_bytes�[0m�[2m=�[0m644245092 �[3mreason�[0m�[2m=�[0mSome(ServerMemoryExhausted)
Evidence: WebSocket driver and launchers 
// Drives a running inputlayer-server over /ws and prints a transcript.
// usage: node drive.mjs <port> <scenario>
const [port, scenario] = process.argv.slice(2);
const PASSWORD = process.env.PW || "memtest-pw";
const NODES = Number(process.env.NODES || 1500);
const CLOSURE = "reach(X, Y) <- edge(X, Y)\nreach(X, Z) <- reach(X, Y), edge(Y, Z)\n?reach(X, Y)";

function edges(from, to) {
  const out = [];
  for (let i = from; i < to; i++)
    for (const j of [(i + 1) % NODES, (2 * i) % NODES, (3 * i + 1) % NODES]) out.push(`(${i}, ${j})`);
  return `+edge[${out.join(", ")}]`;
}

class Client {
  constructor(kg) { this.kg = kg; this.queue = []; this.waiters = []; }
  async open() {
    this.ws = new WebSocket(`ws://127.0.0.1:${port}/ws?kg=${this.kg}`);
    this.ws.onmessage = (m) => {
      const f = JSON.parse(m.data);
      const push = f.id === undefined && !["authenticated", "error"].includes(f.type);
      if (push) return;
      const w = this.waiters.shift();
      if (w) w(f); else this.queue.push(f);
    };
    await new Promise((r, j) => { this.ws.onopen = r; this.ws.onerror = j; });
    this.ws.send(JSON.stringify({ type: "login", username: "admin", password: PASSWORD }));
    const a = await this.reply();
    if (a.type !== "authenticated") throw new Error(JSON.stringify(a));
    return this;
  }
  reply() { return this.queue.length ? Promise.resolve(this.queue.shift()) : new Promise((r) => this.waiters.push(r)); }
  async exec(id, program, show = true) {
    const t = Date.now();
    this.ws.send(JSON.stringify({ type: "execute", id, program }));
    const r = await this.reply();
    const ms = Date.now() - t;
    if (show) console.log(`>> [${id}] ${program.length > 120 ? program.slice(0, 120) + "...(" + program.length + " chars)" : program.replace(/\n/g, " \\n ")}\n<< (${ms} ms) ${summarize(r)}`);
    return r;
  }
}

function summarize(r) {
  const c = { ...r };
  if (Array.isArray(c.rows) && c.rows.length > 5) c.rows = [...c.rows.slice(0, 5), `...${c.rows.length - 5} more`];
  return JSON.stringify(c);
}

async function rss() {
  // RSS of the server process (pid from env), in MiB
  const pid = process.env.SERVER_PID;
  if (!pid) return "n/a";
  const fs = await import("node:fs");
  try {
    const s = fs.readFileSync(`/proc/${pid}/status`, "utf8");
    const m = s.match(/VmRSS:\s+(\d+)/); const h = s.match(/VmHWM:\s+(\d+)/);
    return `RSS ${(m[1] / 1024).toFixed(0)} MiB, peak ${(h[1] / 1024).toFixed(0)} MiB`;
  } catch { return "n/a"; }
}

const check = (cond, msg) => { console.log(`${cond ? "PASS" : "FAIL"}: ${msg}`); if (!cond) process.exitCode = 1; };
const errCode = (r) => r.type === "error" ? r.code : (r.errors && r.errors[0] && r.errors[0].code);

async function setup(kg) {
  const admin = await new Client("default").open();
  await admin.exec(`create-${kg}`, `.kg create ${kg}`);
  const c = await new Client(kg).open();
  return c;
}

const scenarios = {
  async runaway() {
    const c = await setup("runaway");
    const w = await c.exec("load", edges(0, NODES));
    check(w.type === "result", "1500-node edge graph (4500 facts) loads");
    console.log("server before closure:", await rss());
    const r = await c.exec("closure", CLOSURE);
    check(errCode(r) === "resource_exhausted", "unbound transitive closure refused with resource_exhausted");
    check(/max_query_memory_bytes/.test(r.message || "") && /nothing was applied/.test(r.message || ""), "message names the limit and says nothing was applied");
    console.log("server after closure:", await rss());
    const b = await c.exec("bound", "?edge(0, Y)");
    check(b.type === "result" && b.row_count > 0, "same connection still answers a small query");
    const s = await c.exec("small-closure", "reach(X, Y) <- edge(X, Y)\nreach(X, Z) <- reach(X, Y), edge(Y, Z)\n?reach(0, 1)");
    check(s.type === "result", "bounded closure from one node still answers");
  },
  async compact() {
    const c = await setup("compact");
    await c.exec("load", edges(0, NODES));
    await c.exec("rules", "+reach(X, Y) <- edge(X, Y)\n+reach(X, Z) <- reach(X, Y), edge(Y, Z)\n+spare(X) <- edge(X, 0)");
    const before = await c.exec("count-before", "?edge(X, Y)", false);
    console.log(`edge rows before: ${before.row_count}`);
    const q = await c.exec("compact-query", `.compact\n${CLOSURE}`);
    check(errCode(q) === "resource_exhausted" && q.errors[0].index === 3, ".compact then closure query: query statement refused resource_exhausted");
    const p = await c.exec("compact-why", ".compact\n.why ?reach(X, Y)");
    check(errCode(p) === "resource_exhausted", ".compact then .why proof: refused resource_exhausted (not validation)");
    const d = await c.exec("cond-delete", ".rule drop spare\n-edge(X, Y) <- reach(X, Y)");
    check(errCode(d) === "resource_exhausted", "rule drop then conditional delete over closure: refused resource_exhausted (not validation)");
    const after = await c.exec("count-after", "?edge(X, Y)", false);
    console.log(`edge rows after: ${after.row_count}`);
    check(after.row_count === before.row_count, "conditional delete applied nothing");
    console.log("server:", await rss());
  },
  async graphbudget() {
    const c = await setup("budget");
    const a = await c.exec("small", edges(0, 100));
    check(a.type === "result", "small write within 64 KiB graph budget accepted");
    const n0 = (await c.exec("n0", "?edge(X, Y)", false)).row_count;
    console.log(`edge rows: ${n0}`);
    const b = await c.exec("big", edges(100, 500));
    check(errCode(b) === "resource_exhausted" && /max_graph_memory_bytes/.test(b.message || ""), "write past graph budget refused with resource_exhausted naming max_graph_memory_bytes");
    const n1 = (await c.exec("n1", "?edge(X, Y)", false)).row_count;
    check(n1 === n0, `nothing applied (rows ${n1})`);
    const p = await c.exec("program", `+note(1)\n${edges(100, 500)}`);
    check(errCode(p) === "resource_exhausted" && p.errors[0].index === 1, "program: inserting statement refused");
    const note = await c.exec("note", "?note(X)");
    check((note.row_count || 0) === 0, "earlier +note in the same program not applied");
    const del = await c.exec("delete", "-edge(X, Y) <- edge(X, Y)");
    check(del.type === "result", "delete accepted while near budget");
    const again = await c.exec("again", edges(100, 200));
    check(again.type === "result", "freed space accepts new facts");
    // other graph unaffected by this graph's budget usage
    const other = await setup("budget2");
    const o = await other.exec("other", edges(0, 100));
    check(o.type === "result", "a second graph has its own budget");
  },
  async pool() {
    // several runaway closures at once against a small global pool; small queries still pass
    const clients = [];
    const c0 = await setup("pool");
    await c0.exec("load", edges(0, NODES));
    for (let i = 0; i < 4; i++) clients.push(await new Client("pool").open());
    const t = Date.now();
    const rs = await Promise.all(clients.map((c, i) => c.exec(`closure-${i}`, CLOSURE)));
    console.log(`4 concurrent closures finished in ${Date.now() - t} ms`);
    check(rs.every((r) => errCode(r) === "resource_exhausted"), "every concurrent runaway closure refused with resource_exhausted");
    console.log("messages:", rs.map((r) => r.message).join(" | "));
    console.log("server:", await rss());
    const b = await c0.exec("bound", "?edge(0, Y)");
    check(b.type === "result", "server still answers after concurrent runaways");
  },
  async info() {
    const c = await setup("info");
    await c.exec("load", edges(0, NODES));
    console.log("server before:", await rss());
    const r = await c.exec("closure", CLOSURE);
    check(errCode(r) === "resource_exhausted", "closure refused under container-derived pool");
    console.log("message:", r.message);
  },
};

try { await scenarios[scenario](); } catch (e) { console.log("FAIL: exception", e); process.exitCode = 1; }
process.exit();

Pipeline

Updates from git push no-mistakes

... (3 earlier update rounds omitted to keep the PR body within GitHub's 65536-char limit; full history is in the run log.)

🔧 **Review** - 2 issues found → auto-fixed (4) ✅

🔧 Fix applied.
3 warnings still open:

  • ⚠️ deploy/kubernetes/base/configmap.yaml:28 - The shipped deployment limits refuse ordinary queries over a graph that is still inside its own budget. The k8s base config sets max_query_memory_bytes = 512 MiB with max_graph_memory_bytes = 1 GiB. docs/content/docs/guides/deployment.mdx:78-79 sets 1 GiB per query against 4 GiB per graph. Query evaluation copies its input onto the metered compute thread: StaticSource::collect clones every matching tuple into a Vec before to_stream (src/code_generator/static_input.rs), and DD batches and arrangements copy the data again. The new unit test test_memory_limit_stops_the_computation confirms that a plain Scan is charged. Example: a graph of about 5M two-int tuples is about 600 MB estimated, which is under the 1 GiB budget. An unfiltered ?edge(X, Y) or a join over it holds more than 512 MiB and gets resource_exhausted, even though nothing runs away. Either raise the per-query limit relative to the graph budget in these shipped examples, or document that queries scanning a whole graph need max_query_memory_bytes of roughly 1–2× the graph budget. The values are an operator-facing product decision.
  • ⚠️ deploy/kubernetes/base/configmap.yaml:31 - The fix round raised the shipped per-query limit to 2 GiB (the user asked for at least 2x the graph budget). With that value, the k8s base config breaks its own sizing rule at lines 25-27 and can no longer keep the process inside its container. statefulset.yaml:119 limits the pod to 4Gi and sets no CPU limit. Concurrency comes from the query semaphore, which allows ncpu - max(ncpu/4, 2) queries at once (src/protocol/handler.rs:830-840), and available_parallelism reports all of the node's cores. Concrete case on a 4-core node: 2 permits. Two runaway queries can each hold just under 2 GiB, and one graph holds 1 GiB. That totals more than 5 GiB, so the pod is OOM-killed before either query hits resource_exhausted. This contradicts the intent that 'the process must stay inside its container limit so it is not killed'. Larger nodes are worse: a 16-core node allows 12 permits, or 24 GiB of query memory. The old 512 MiB value already broke the rule on large nodes, but the fix round made it break on every typical node. The same problem appears in docs/content/docs/guides/deployment.mdx:78-79, which allows 8 GiB per query on all cores while its sizing table (around line 566) recommends 8-16 GB for medium production. Possible remedies, all involving product values: lower the graph budget (for example 512 MiB graph and 1 GiB query), raise the container memory limit, or cap concurrent compute permits through config so the documented sum holds. That choice is the author's.
  • ⚠️ src/code_generator/mod.rs:1191 - The per-query limit is enforced per worker thread, not per query. In the multi-worker path, each rayon worker installs its own (control, thread_net_bytes()) baseline and charges only its own thread's growth against memory_limit. A query running on N workers can therefore hold about N x max_query_memory_bytes before any worker stops. This path is reachable in documented usage: deploy/kubernetes/base/configmap.yaml:21-22 tells operators to set num_threads to the requested CPUs, and num_threads becomes the KG's num_workers (src/storage_engine/mod.rs:340). Concrete case: num_threads = 8 and max_query_memory_bytes = 2 GiB. A non-recursive scan or join over a large graph, split into 8 partitions, can hold about 16 GiB. Yet deployment.mdx and the configmap describe max_query_memory_bytes as bounding 'a query's working set' and size the container per query. Remedy: charge each worker's growth to a shared per-request counter in RequestControl, for example an AtomicI64 of held bytes that each thread updates by its delta, and compare the sum against the limit. Alternatively, document the limit as per-worker and scale the sizing formula by num_threads. Which semantics to use is the author's call.

🔧 Fix applied.
4 warnings still open:

  • ⚠️ deploy/kubernetes/base/configmap.yaml:28 - The shipped deployment limits refuse ordinary queries over a graph that is still inside its own budget. The k8s base config sets max_query_memory_bytes = 512 MiB with max_graph_memory_bytes = 1 GiB. docs/content/docs/guides/deployment.mdx:78-79 sets 1 GiB per query against 4 GiB per graph. Query evaluation copies its input onto the metered compute thread: StaticSource::collect clones every matching tuple into a Vec before to_stream (src/code_generator/static_input.rs), and DD batches and arrangements copy the data again. The new unit test test_memory_limit_stops_the_computation confirms that a plain Scan is charged. Example: a graph of about 5M two-int tuples is about 600 MB estimated, which is under the 1 GiB budget. An unfiltered ?edge(X, Y) or a join over it holds more than 512 MiB and gets resource_exhausted, even though nothing runs away. Either raise the per-query limit relative to the graph budget in these shipped examples, or document that queries scanning a whole graph need max_query_memory_bytes of roughly 1–2× the graph budget. The values are an operator-facing product decision.
  • ⚠️ deploy/kubernetes/base/configmap.yaml:31 - The fix round raised the shipped per-query limit to 2 GiB (the user asked for at least 2x the graph budget). With that value, the k8s base config breaks its own sizing rule at lines 25-27 and can no longer keep the process inside its container. statefulset.yaml:119 limits the pod to 4Gi and sets no CPU limit. Concurrency comes from the query semaphore, which allows ncpu - max(ncpu/4, 2) queries at once (src/protocol/handler.rs:830-840), and available_parallelism reports all of the node's cores. Concrete case on a 4-core node: 2 permits. Two runaway queries can each hold just under 2 GiB, and one graph holds 1 GiB. That totals more than 5 GiB, so the pod is OOM-killed before either query hits resource_exhausted. This contradicts the intent that 'the process must stay inside its container limit so it is not killed'. Larger nodes are worse: a 16-core node allows 12 permits, or 24 GiB of query memory. The old 512 MiB value already broke the rule on large nodes, but the fix round made it break on every typical node. The same problem appears in docs/content/docs/guides/deployment.mdx:78-79, which allows 8 GiB per query on all cores while its sizing table (around line 566) recommends 8-16 GB for medium production. Possible remedies, all involving product values: lower the graph budget (for example 512 MiB graph and 1 GiB query), raise the container memory limit, or cap concurrent compute permits through config so the documented sum holds. That choice is the author's.
  • ⚠️ src/protocol/handler.rs:2510 - A memory stop after a commit is reported as validation everywhere except a plain query. Round 1 (334b0c9) made memory limits apply to computations after the request began committing, but it remapped the error code only in QueryJob (src/protocol/handler.rs:3430). Two sibling paths that also compute after a durable statement still use a hard-coded ErrorCode::Validation: .why proof search (run_proof!, handler.rs:2510) and fact staging for conditional delete and update (src/protocol/handler/fact_staging.rs:51, applied at :91 and :103 through invalid, then passed through by write_run.rs:236). Example: the program .compact followed by -edge(X, Y) &lt;- reach(X, Y) over a large closure. .compact enters COMMITTING. Staging's evaluate then hits the per-query limit, so charge_memory sets memory_exceeded. supervise's finish() returns Ok because the request is committing, so nothing overrides the code, and the statement fails with code: validation instead of resource_exhausted. The same happens for .compact followed by .why ?reach(X, Y), and for ServerMemoryExhausted. Fix: derive the code once at a shared point. For example, add a helper that returns ResourceExhausted when current_request_control().memory_exceeded().is_some(), and use it in QueryJob, run_proof!, and the staging error mapping, so all three sites agree.
  • ⚠️ deploy/kubernetes/base/configmap.yaml:24 - The k8s sizing written in the round-2 fix leaves no headroom, so the pod can still be OOM-killed while every limit holds. The configmap and deployment.mdx (around line 600) budget 2 GiB of queries (the derived pool, half of 4Gi) plus 3 × 512 MiB of graphs plus 0.5 GiB of server, which is exactly the 4Gi pod limit. All three terms undercount real memory. (1) MeteredAllocator counts requested bytes (memory.rs:161-195), not RSS. glibc malloc headers, size-class rounding and per-thread arena retention add substantial overhead for DD's many small Tuple/Vec allocations, so 2 GiB metered is noticeably more resident. (2) max_graph_memory_bytes is an estimate (Tuple::estimated_bytes plus 32 bytes of index). It leaves out snapshot copy-on-write chunks that running queries keep pinned while writes or deletes rebuild a relation (data.retain in relation_store.rs). (3) Queries only charge at checkpoints, so they can overshoot between them. Concrete case: two queries near 1 GiB metered each, plus three graphs at budget, plus one delete rebuilding a pinned relation, exceeds 4Gi of RSS and the pod is killed. That contradicts the intent that 'the process must stay inside its container limit'. The base config also depends on detecting the cgroup at runtime for the pool (container_memory_limit returns None if /sys/fs/cgroup/memory.max is not the container's own cgroup), and then the pool silently becomes unlimited. Remedies involve product values: keep explicit headroom in the sizing rule (for example 20–25%), set max_total_query_memory_bytes explicitly in the configmap, lower the number of graphs the rule assumes, or raise the pod limit. The choice is the author's.

🔧 Fix applied.
5 issues (4 warnings, 1 info) still open:

  • ⚠️ deploy/kubernetes/base/configmap.yaml:28 - The shipped deployment limits refuse ordinary queries over a graph that is still inside its own budget. The k8s base config sets max_query_memory_bytes = 512 MiB with max_graph_memory_bytes = 1 GiB. docs/content/docs/guides/deployment.mdx:78-79 sets 1 GiB per query against 4 GiB per graph. Query evaluation copies its input onto the metered compute thread: StaticSource::collect clones every matching tuple into a Vec before to_stream (src/code_generator/static_input.rs), and DD batches and arrangements copy the data again. The new unit test test_memory_limit_stops_the_computation confirms that a plain Scan is charged. Example: a graph of about 5M two-int tuples is about 600 MB estimated, which is under the 1 GiB budget. An unfiltered ?edge(X, Y) or a join over it holds more than 512 MiB and gets resource_exhausted, even though nothing runs away. Either raise the per-query limit relative to the graph budget in these shipped examples, or document that queries scanning a whole graph need max_query_memory_bytes of roughly 1–2× the graph budget. The values are an operator-facing product decision.
  • ⚠️ deploy/kubernetes/base/configmap.yaml:31 - The fix round raised the shipped per-query limit to 2 GiB (the user asked for at least 2x the graph budget). With that value, the k8s base config breaks its own sizing rule at lines 25-27 and can no longer keep the process inside its container. statefulset.yaml:119 limits the pod to 4Gi and sets no CPU limit. Concurrency comes from the query semaphore, which allows ncpu - max(ncpu/4, 2) queries at once (src/protocol/handler.rs:830-840), and available_parallelism reports all of the node's cores. Concrete case on a 4-core node: 2 permits. Two runaway queries can each hold just under 2 GiB, and one graph holds 1 GiB. That totals more than 5 GiB, so the pod is OOM-killed before either query hits resource_exhausted. This contradicts the intent that 'the process must stay inside its container limit so it is not killed'. Larger nodes are worse: a 16-core node allows 12 permits, or 24 GiB of query memory. The old 512 MiB value already broke the rule on large nodes, but the fix round made it break on every typical node. The same problem appears in docs/content/docs/guides/deployment.mdx:78-79, which allows 8 GiB per query on all cores while its sizing table (around line 566) recommends 8-16 GB for medium production. Possible remedies, all involving product values: lower the graph budget (for example 512 MiB graph and 1 GiB query), raise the container memory limit, or cap concurrent compute permits through config so the documented sum holds. That choice is the author's.
  • ⚠️ deploy/kubernetes/base/configmap.yaml:24 - The k8s sizing written in the round-2 fix leaves no headroom, so the pod can still be OOM-killed while every limit holds. The configmap and deployment.mdx (around line 600) budget 2 GiB of queries (the derived pool, half of 4Gi) plus 3 × 512 MiB of graphs plus 0.5 GiB of server, which is exactly the 4Gi pod limit. All three terms undercount real memory. (1) MeteredAllocator counts requested bytes (memory.rs:161-195), not RSS. glibc malloc headers, size-class rounding and per-thread arena retention add substantial overhead for DD's many small Tuple/Vec allocations, so 2 GiB metered is noticeably more resident. (2) max_graph_memory_bytes is an estimate (Tuple::estimated_bytes plus 32 bytes of index). It leaves out snapshot copy-on-write chunks that running queries keep pinned while writes or deletes rebuild a relation (data.retain in relation_store.rs). (3) Queries only charge at checkpoints, so they can overshoot between them. Concrete case: two queries near 1 GiB metered each, plus three graphs at budget, plus one delete rebuilding a pinned relation, exceeds 4Gi of RSS and the pod is killed. That contradicts the intent that 'the process must stay inside its container limit'. The base config also depends on detecting the cgroup at runtime for the pool (container_memory_limit returns None if /sys/fs/cgroup/memory.max is not the container's own cgroup), and then the pool silently becomes unlimited. Remedies involve product values: keep explicit headroom in the sizing rule (for example 20–25%), set max_total_query_memory_bytes explicitly in the configmap, lower the number of graphs the rule assumes, or raise the pod limit. The choice is the author's.
  • ⚠️ deploy/kubernetes/base/configmap.yaml:26 - Round 3 added a sizing rule that does not keep the process inside its container, which the intent requires ('the process must stay inside its container limit so it is not killed'). The rule is: max_total_query_memory_bytes <= 0.6 × pod limit, and the sum of max_graph_memory_bytes <= max_total_query_memory_bytes. Stored graph facts are not part of the query pool. The pool only counts what computation threads hold while a request runs, and RequestControl::drop returns that to the pool when the request ends (src/execution/request_control.rs:372). So the two budgets add up: queries 0.6 + graphs up to 0.6 = 1.2 × the pod limit, before any allocator overhead. Concrete case, with the k8s base as shipped: an 8Gi pod has a derived pool of about 4.8 GiB, and the configmap itself says 'Up to four graphs at their full budget fit the rule' at 1 GiB each. That is 8.8 GiB of metered bytes, so the pod is OOM-killed while every limit holds. The deployment.mdx example (around line 600) has the same problem: on a 32 GB host, 16 GiB of queries plus four graphs × 4 GiB is the whole host with no headroom. Yet the docs say 'The remaining 40% of the memory limit is headroom', which is false. The same wording appears in config.example.toml:98-102, docs/content/docs/guides/configuration.mdx:138-142, docs/guides/configuration.md and gui/lib/docs-bundle.ts. The rule matches the user's round-3 instruction word for word, so the fix needs a product decision. One option: make queries and graphs share the 60%, i.e. total query budget + sum of graph budgets <= 0.6 × limit (for an 8Gi pod, about 2.8 GiB of queries with four 512 MiB graphs, or set max_total_query_memory_bytes explicitly). Another: lower the derived default so it leaves room for graphs. Either way, fix the 'four graphs fit' and '40% headroom' claims.
  • ℹ️ src/protocol/handler/fact_staging.rs:96 - Round 3 remapped the post-commit memory stop on the fact-staging path (conditional delete and update, fact_staging.rs:96 and :107) to resource_exhausted. The round's test only covers .why (tests/memory_limit_tests.rs a_proof_after_a_commit_is_refused_as_resource_exhausted). No test runs .compact followed by -edge(X, Y) &lt;- reach(X, Y) over a large closure, so a regression back to validation on this path would go unnoticed. A single integration case like the .why one would cover it.

🔧 Fix applied.
1 info still open:

  • ℹ️ src/protocol/handler/fact_staging.rs:96 - Round 3 remapped the post-commit memory stop on the fact-staging path (conditional delete and update, fact_staging.rs:96 and :107) to resource_exhausted. The round's test only covers .why (tests/memory_limit_tests.rs a_proof_after_a_commit_is_refused_as_resource_exhausted). No test runs .compact followed by -edge(X, Y) &lt;- reach(X, Y) over a large closure, so a regression back to validation on this path would go unnoticed. A single integration case like the .why one would cover it.

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 7 of 7 scenarios driven live against the product
Scenario Result Live Evidence
A runaway unbound transitive-closure query with a 32 MiB per-query limit is refused with resource_exhausted in under 0.3 s, and the same connection then answers bound and small queries ✅ pass live live_query_limit.txt: code resource_exhausted citing max_query_memory_bytes; RSS 77 MiB; bound query returns 2 rows, small closure returns 7
.compact followed by .why ?reach(X,Y) over a large closure reports resource_exhausted, not validation ✅ pass live live_query_limit.txt: errors[0] index 1, code resource_exhausted
A rule drop followed by the conditional delete -edge(X,Y) &lt;- reach(X,Y) over a large closure is refused as resource_exhausted and deletes nothing ✅ pass live live_query_limit.txt: errors[0] code resource_exhausted, 'none of the program's 2 write statements was applied'; edge count stays 4496
A write that would grow a graph past max_graph_memory_bytes is refused, nothing is applied (including in a multi-statement program), and deletes still free space for new writes ✅ pass live live_graph_budget.txt: oversized write gets resource_exhausted citing max_graph_memory_bytes; count stays 298; note(1) not applied; delete removes 298 facts; re-insert of 300 succeeds
Concurrent queries that each fit but together exceed max_total_query_memory_bytes are refused with resource_exhausted, and the server keeps serving ✅ pass live live_global_pool.txt: both concurrent closures get resource_exhausted citing max_total_query_memory_bytes; bound query afterwards succeeds
Adversarial: in a 512M memory-limited container with no explicit limits, the pool defaults to 60% of the cgroup limit, and a runaway query that needs 1131 MiB unlimited is refused instead of the proce… ✅ pass live live_cgroup_default_pool.txt (startup log total_query_memory_bytes=322122546 = 0.6 x 536870912; resource_exhausted; peak 411 MiB; bound query works) vs live_baseline_unlimited.txt (1131 MiB peak)
The server starts with the config shipped in the Kubernetes configmap and logs the explicit query limits (1 GiB per query, 3 GiB total), which fit the documented 0.8 x 8Gi sizing rule ✅ pass live live_k8s_configmap_startup.txt
  • cargo build --release --bin inputlayer-server
  • cargo test --release --test memory_limit_tests (6 real-WebSocket integration tests: runaway query, runaway after commit, .why after commit, conditional delete after commit, query under the limit, graph budget)
  • Live: inputlayer-server --config <tmp> with max_query_memory_bytes=32MiB, driven by a Node WebSocket client: unbound closure over 4496 edges, bound query, small query, .compact + .why ?reach(X,Y), .rule drop spare + -edge(X,Y) &lt;- reach(X,Y), edge count afterwards
  • Live: max_graph_memory_bytes=64KiB: small write, oversized write, +note(1) program with an oversized write, delete-all, re-insert
  • Live: max_query_memory_bytes=0, max_total_query_memory_bytes=48MiB: two concurrent closure queries on separate connections, then a bound query
  • Live baseline: everything unlimited, same runaway closure, peak RSS measured (1131 MiB)
  • Live: server under systemd-run --user --scope -p MemoryMax=512M + unshare -Urm -C with its own cgroup2 mounted at /sys/fs/cgroup, no explicit limits: startup log shows total_query_memory_bytes=322122546 (60%), runaway closure refused, process survives (peak 411 MiB)
  • Live: server started with config.toml taken from deploy/kubernetes/base/configmap.yaml; startup log shows the 1 GiB per-query and 3 GiB total limits; listens normally

✅ No issues found.

  • Live validation: ✅ go - 6 of 7 scenarios driven live against the product
Scenario Result Live Evidence
Tenant runs an unbound transitive closure over a 1500-node graph with max_query_memory_bytes=32MiB: gets resource_exhausted naming the limit and 'nothing was applied'; the same connection then answers… ✅ pass live s1-runaway.txt (refused in 299 ms, server RSS 78 MiB, follow-up queries return results)
Runaway computation after the program began committing (.compact + closure query, .compact + .why, rule drop + conditional delete over closure) is reported as resource_exhausted, not validation, and t… ✅ pass live s2-after-commit.txt (all three error code resource_exhausted; edge rows 4496 before and after)
Write that would grow a graph past max_graph_memory_bytes=64KiB is refused with resource_exhausted and applies nothing, including earlier statements in the same program; deletes are accepted and free… ✅ pass live s3-graph-budget.txt
Adversarial: four concurrent runaway closures, each under its 1 GiB per-query limit, are all refused once together they exceed max_total_query_memory_bytes=64MiB; the server still serves afterwards ✅ pass live s4-global-pool.txt (4x resource_exhausted naming max_total_query_memory_bytes, ServerMemoryExhausted in log)
Process stays inside its container: in a 1 GiB Docker container with shipped defaults (no explicit pool), the pool derives to 60% of the cgroup limit, the runaway is refused and the container is not O… ✅ pass live s5b-container-shipped-defaults.txt (pool_budget_bytes=644245092 = 0.6 x 1GiB, memory.peak 632 MiB, oom_kill 0, running=true)
Baseline contrast: same container and query with both limits disabled is OOM-killed, reproducing the original incident the guard prevents ✅ pass live s5a-container-limits-disabled.txt (OOMKilled=true exit=137, startup warns max_query_memory_bytes=0)
A query under the limit is unaffected (closure over a tiny graph returns all 6 rows) ⏸️ untested no The prior payload did not establish a live result for this scenario. The exact 6-row case was run only as the in-process integration test a_query_under_the_limit_is_unaffected, not against the running…
  • cargo build --release --bin inputlayer-server
  • run.sh runaway - fresh server with max_query_memory_bytes=32MiB; unbound transitive closure over a 1500-node graph; follow-up small and bounded queries on the same connection
  • run.sh compact - .compact+closure query, .compact+.why ?reach(X,Y), .rule drop spare+-edge(X,Y) &lt;- reach(X,Y); edge count before and after
  • run.sh graphbudget - max_graph_memory_bytes=64KiB: small write, oversized write, mixed program, delete, re-insert, second graph
  • run.sh pool - per-query 1GiB, max_total_query_memory_bytes=64MiB, 4 concurrent runaway closures, then a small query
  • run-container.sh 18740 1g with both limits set to 0 (contrast) - Docker ubuntu:24.04 with --memory=1g, 3000-node closure
  • run-container.sh 18741 1g with shipped defaults (pool unset, derived from cgroup memory.max) - same 3000-node closure
  • cargo test --test memory_limit_tests (6 integration tests over /ws)
  • cargo test --lib test_total_query_memory_defaults_to_60_percent_of_the_cgroup_limit
✅ **Document** - passed

✅ No issues found.

✅ No issues found.

🔧 **Lint** - 1 issue found ✅
  • ℹ️ deploy/kubernetes/base/configmap.yaml - make k8s-check (scripts/check-k8s-manifests.sh) could not run because kubectl is not on PATH, so the changed k8s manifests (configmap.yaml, statefulset.yaml with the 8Gi limit) were not validated. ruff is also not installed (and uv is offline), so the Python docstring was checked for line length by hand only.

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

✅ No issues found.

@jsam jsam mentioned this pull request Oct 4, 2026
Closed
@jsam jsam changed the title feat(engine): refuse queries over a per-query memory limit and writes over a per-graph budget with resource_exhausted (EN-9) feat: refuse queries and writes over memory limits with resource_exhausted Oct 4, 2026
jsam added 6 commits October 4, 2026 19:59
@jsam
feat(engine): refuse queries over a per-query memory limit and writes…
d64c12f 
… over a per-graph memory budget with resource_exhausted

A query that grows past storage.performance.max_query_memory_bytes
(default 4 GiB) is stopped and refused with the new wire code
resource_exhausted instead of growing the process; a write that would grow
a knowledge graph's estimated fact bytes past
storage.performance.max_graph_memory_bytes (default unlimited) is refused
the same way. Nothing either would have changed is applied, deletes always
pass, and recovery loads a graph over budget.

The global allocator meters each thread's net allocation in a thread-local
cell (no lock, no atomic), and the evaluator charges it to the request's
RequestControl at its cooperative checkpoints. Recursive loops now check at
every batch (one step of an iterative scope can run a whole fixpoint), and a
stopped dataflow is dropped instead of being run to completion by
execute_directly, so deadlines and cancels also stop recursive queries
promptly.
@jsam
no-mistakes(review): Enforce query memory limit after commits; raise …
20b8a34 
…shipped per-query limits
@jsam
no-mistakes(review): Enforce memory limits per query and across concu…
14afa87 
…rrent queries
@jsam
no-mistakes(review): Report memory stops as resource_exhausted everyw…
e0d9d9c 
…here; 60% cgroup budget
@jsam
no-mistakes(review): Correct memory sizing rule; test conditional del…
d4d2f71 
…ete memory stop
@jsam
no-mistakes(document): Fix rustdoc private link and rewrap error-code…
338e6cb 
… doc comments
@jsam jsam changed the title feat: refuse queries and writes over memory limits with resource_exhausted feat(execution): refuse queries and writes over memory limits with resource_exhausted Oct 4, 2026
@jsam
jsam marked this pull request as ready for review October 4, 2026 20:20
@jsam
jsam merged commit 26565e7 into main Oct 4, 2026
21 checks passed
@jsam
jsam deleted the fm/en-en9 branch October 4, 2026 20:21
This was referenced Oct 4, 2026
Closed
Merged
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

See More

Successfully merging this pull request may close these issues.

1 participant

@jsam