Repository navigation
fix(storage): load previously resident knowledge graphs in the background after a restart - #389
Merged
Merged
Conversation
…ckground after a restart (#373) Since #357 a restart reads metadata only and each knowledge graph loads on first use, so the first query after a restart paid the load. The engine now keeps the names of the loaded graphs in metadata/resident_knowledge_graphs.json (rewritten when a graph loads that it lacks, is unloaded or is dropped), and the server loads those graphs in a background task right after startup. Startup and logins do not wait for it, graphs that were not in memory stay dormant, and storage.max_loaded_knowledge_graphs still bounds what is loaded.
…mit from the resident record A graph skipped because storage.max_loaded_knowledge_graphs was reached stayed recorded as in memory, so the next restart could warm stale names instead of the graphs that were loaded.
… knowledge graphs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
Fix GitHub issue #373 in inputlayer/inputlayer (production readiness, performance): "First query after restart is 5x slower (3.7 -> 18.9 ms) since lazy graph loading (#357)".
Found by the readiness benchmark checkpoint on the benchmark host (2026-10-05, report in the business repo docs/performance). The first query now pays the graph load. Restart+first query 121 ms vs 105 ms before. Options: optional prewarm of recently used graphs at startup (config), or background load right after startup without blocking login. Acceptance: first query after restart within 10% of before, restart-to-login unchanged.
Deliver it as a small PR through the merge train; done means merged with CI green.
What Changed
metadata/resident_knowledge_graphs.json. It rewrites the file when a graph loads that the file lacks, or when one is created, unloaded or dropped. On open it reads the file and discards names that are no longer registered; a missing or unreadable file reads as empty.Handler::warm_knowledge_graphs) that loads each recorded graph in turn and logskg_warm_complete. Startup and logins do not wait for it, and a request that arrives before the warm-up reaches its graph loads it itself.max_loaded_knowledge_graphsand never unload another graph (activategains atrimflag). A graph left dormant by the limit is removed from the record until a request loads it. Four tests inresidency_tests.rscover this and the restart behaviour, and the configuration and persistence docs describe the background load.🤖 Generated with Claude Code
Measurement (benchmark host, against the acceptance line)
recoveryfixture of the perf-gate engine suite onsam-dev-benchmarks(10K edges, a two-hop rule and 50K durable facts; SIGKILL, restart, login, first?two_hop(1, Z)), 10 rounds of 3 restarts per arm, each run paired against a6c20e3, the commit before #357. Values are the range of per-round p50.20261005T211232Z-32b56c62,20261005T213816Z-dc203d65); the "before" column is from the fix's run. The numbers were taken on dc203d6, the first commit of this branch; the later commits change only themax_loaded_knowledge_graphspath, which the fixture does not set.Option chosen: background load right after startup without blocking login, rather than a configured prewarm. It meets the acceptance line and needs no new operator configuration.
Lazy loading's benefit (#357) kept, and how it was checked: startup still reads metadata only (the load runs in a background task after the handler is built; restart-to-login above is unchanged), and only the graphs that were in memory before the restart are loaded, never more than
max_loaded_knowledge_graphs. The benchmark fixture has one user graph, so the evidence that other graphs stay dormant is the unit tests inresidency_tests.rs(a_restart_leaves_unloaded_and_dropped_knowledge_graphs_dormant,warming_stops_at_the_loaded_limit_and_survives_a_bad_record). With the default never-unload settings a graph used once stays in the reload set across restarts;unload_idle_after_secsdrops idle ones.Risk Assessment
✅ Low: The fix round's change is a single
trimflag onactivatethat makes the warm path skipunload_over_limitas the user decided, request loads still trim, the doc comment matches the behaviour, and the new race test fails against the pre-fix code; the rest of the warm-up is a bounded background load that leaves startup and login paths unchanged.Testing
Built release servers from the target commit and from the base commit, and drove both over the real WebSocket login/query path on disposable data directories with a Node driver. Covered: first-query and restart-to-login timing across 15 alternating restarts per arm at two graph sizes; a 3M-fact graph to show login does not wait and a racing query is answered correctly with a single load; dormant graphs staying dormant; the loaded-graph limit; corrupt, empty and stale resident records; a dropped graph; and a SIGKILL restart. Also ran the 21 residency unit tests. Everything passed; transcripts are in the evidence directory, and test data and binaries were removed from the worktree. No UI surface is involved, so there are no screenshots.
kg_activated big elapsed_ms=1431logged 1379 ms after the login; first query on it afterwards 13.99 ms with the right rowbigwas loading returned [[1111111,1111112]] after 1306 ms, exactly onekg_activatedforbig; first-query.txt (200k facts): racing first query 14.58 ms…aloads it, answers [[5,6]] and adds it to the recorddrecord is ["_internal","d"]; next restart warms exactly _internal and d, unloaded []kg_warm_failed, no load attempted forghostor../../etc.kg drop crecord is ["_internal","a","b"]; after SIGKILL restart activated ["_internal","a","b"],cnot loaded,?edge(1999, X)returns 2000Evidence: First query and login after restart at the issue's scale (40k facts): this change vs base commit
medians (ms) this change, warm-up this change, no record base commit a649cd9e restart_to_login 69.02 69.11 68.28 first_query 1.32 13.42 13.25 second_query 1.00 0.73 0.77 login+first_query 70.35 82.53 81.53 warm arm: bench already loaded when the first query was sent in 15/15 restartsEvidence: Same measurement at 200k facts, where the first query races the background load
medians (ms) this change, warm-up this change, no record base commit a649cd9e restart_to_login 67.69 67.30 67.11 first_query 14.58 65.20 65.08 second_query 1.66 1.38 1.47 login+first_query 82.27 132.50 132.19 warm arm: bench already loaded when the first query was sent in 0/15 restartsEvidence: 3M-fact graph: login during the background load, racing query, login time with and without the record
restart a: login on "default" accepted 69.36 ms after spawn; activated so far: ["_internal"] "big" finished loading 1379 ms after the login was accepted first query on "big" after the warm-up: 13.99 ms, rows [[2222222,2222223]] restart b: login on "big" 72.01 ms after spawn, query sent while big still loading: true first query 1305.59 ms rows [[1111111,1111112]]; second 12.35 ms rows [[7,8]] kg_activated events for "big": 1 restart c: restart-to-login median 69.23 ms with the record vs 69.51 ms withoutEvidence: Dormant graphs, loaded-graph limit, bad records, drop and SIGKILL restart
run 2 (restart, no queries): activated ["_internal","c"] run 3 (limit 2, record of 6): activated ["_internal","a"], unloaded []; warmed 2; record now ["_internal","a"] after a query on "d": unloaded ["a"], record ["_internal","d"] run 4 (restart, limit 2): activated ["_internal","d"], unloaded [] record = garbage bytes: login ok, query rows [[5,6]] after SIGKILL restart: activated ["_internal","a","b"] RESULT: all checks passedEvidence: Driver: starts the server and speaks /ws
~/.no-mistakes/evidence/01M474HC10X106Z19KKT9QYWKE/scenarios.mjs)Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
🔧 **Rebase** - 1 issue found → auto-fixed ✅
src/protocol/handler.rs- merge conflict rebasing onto origin/main🔧 Fix applied.
✅ Re-checked - no issues remain.
🔧 **Review** - 1 issue found → auto-fixed ✅
src/storage_engine/residency.rs:587- The recorded decision forwarm-load-can-evict-request-loaded-graph("the warm-up's load must not run unload_over_limit itself") is not present in the target commit.warm_knowledge_graphstill calls the sharedactivate(line 587), which ends withself.unload_over_limit(kg)(line 434) sparing only the warmed graph. The limit check at line 580 is taken once, outside any lock, so the original sequence is still reachable: limit 2, record {_internal, a}; after restart_internalis loaded (count 1); a request starts loadingz(not counted yet); the warm-up reachesa, sees 1 < 2 and loads it;zfinishes (count 2) and its query returns;afinishes (count 3) and itsunload_over_limit("a")unloads idlezand removes it from the record. The next query onzpays a full reload and a restart then warmsainstead ofz. Cost is one extra load per occurrence, no data loss. Same invariant stated in two other places that are currently untrue: the doc comment at src/storage_engine/residency.rs:566 ("the warm-up never unloads a KG a request loaded") and the assertion message at src/storage_engine/residency_tests.rs:701 ("warming never unloads a used graph"), which only exercises the sequential case. Fix as already decided: give the warm path an activation that skipsunload_over_limit(private variant or flag), leaving request-driven loads to trim an over-limit count as today; no configuration.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
kg_activated big elapsed_ms=1431logged 1379 ms after the login; first query on it afterwards 13.99 ms with the right rowbigwas loading returned [[1111111,1111112]] after 1306 ms, exactly onekg_activatedforbig; first-query.txt (200k facts): racing first query 14.58 ms…aloads it, answers [[5,6]] and adds it to the recorddrecord is ["_internal","d"]; next restart warms exactly _internal and d, unloaded []kg_warm_failed, no load attempted forghostor../../etc.kg drop crecord is ["_internal","a","b"]; after SIGKILL restart activated ["_internal","a","b"],cnot loaded,?edge(1999, X)returns 2000cargo build --release --bin inputlayer-server --bin inputlayer-clientat ff27f02e, plus a releaseinputlayer-serverbuilt fromgit archive a649cd9eas the before-binaryIL_BIN=<target server> IL_BASE=<base server> FACTS=40000 ROUNDS=15 node scenarios.mjs <data> first-query— restart, login, first and second query; arms: this change with the record, this change with the record deleted, base commitIL_BIN=<target server> IL_BASE=<base server> FACTS=200000 ROUNDS=15 node scenarios.mjs <data> first-query— same, with a load longer than the loginIL_BIN=<target server> BIG_FACTS=3000000 node scenarios.mjs <data> big— login while a 3M-fact graph loads in the background, a query racing the warm-up, login time with and without the recordIL_BIN=<target server> node scenarios.mjs <data> residency— dormant graphs,INPUTLAYER_STORAGE__MAX_LOADED_KNOWLEDGE_GRAPHS=2, garbage / wrong-shape / unknown-name / emptyresident_knowledge_graphs.json,.kg drop, SIGKILL restartcargo test --lib residency— 21 passedRead the server log for the one 642 ms login outlier:persist_open_complete elapsed_ms=489, before the warm-up is spawneddocs/guides/persistence.md:27- docs/README.md names docs/content/ as the single source of truth, yet this change added the same background-load paragraph to both docs/content/docs/guides/persistence.mdx (owner) and the legacy copy docs/guides/persistence.md. Both copies are accurate today, so nothing was changed here. The whole docs/guides/*.md tree duplicates the MDX guides and is already drifting (docs/guides/configuration.md has no max_loaded_knowledge_graphs entry at all). Proposed follow-up outside this change: delete the legacy docs/guides tree or reduce each file to a pointer to its MDX owner.src/storage_engine/residency.rs:606- The doc comment on record_resident says it is called under the slot's mutex when a KG loads or unloads. That holds for activate, try_unload and the warm-up skip, but create_graph (src/storage_engine/mod.rs:473) and drop (mod.rs:528) call it without that mutex. Left unchanged: the earlier review raised the create_graph site and the user's decision limited the fix to the warm-up skip, so whether to reword the comment or move those calls under the mutex is a human call.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.