Repository navigation
feat(storage): load knowledge graphs on first use so restart reads only metadata - #357
Conversation
…tadata only (EN-13, #135)
# Conflicts: # src/storage_engine/mod.rs
…tion deadlock, per-graph checkpoints
EN-13 measurementsRestart timeMeasured at sam-dev-benchmarks, Engine (
Server (
Loading one 10M-fact KG on first use (1.3 s) is 3.9x faster than main's full startup load of it (5.0 s), because its shards load in parallel. Latency
The same metrics are INCONCLUSIVE in the CI runner's A/A calibration (#198). Post-merge note: after the measured commit, review fixes changed checkpoint capture to hold each dormant graph only while its state is captured (each graph consistent at its own revision), and kept graphs with unsaved catalog changes loaded. Neither touches the startup or first-use load path measured above; the re-run on the merged head is pending until sam-dev-benchmarks is back. |
…ckground after a restart (#373) Since #357 a restart reads metadata only and each knowledge graph loads on first use, so the first query after a restart paid the load. The engine now keeps the names of the loaded graphs in metadata/resident_knowledge_graphs.json (rewritten when a graph loads that it lacks, is unloaded or is dropped), and the server loads those graphs in a background task right after startup. Startup and logins do not wait for it, graphs that were not in memory stay dormant, and storage.max_loaded_knowledge_graphs still bounds what is loaded.
…ound after a restart (#389) * fix(storage): load the knowledge graphs that were in memory in the background after a restart (#373) Since #357 a restart reads metadata only and each knowledge graph loads on first use, so the first query after a restart paid the load. The engine now keeps the names of the loaded graphs in metadata/resident_knowledge_graphs.json (rewritten when a graph loads that it lacks, is unloaded or is dropped), and the server loads those graphs in a background task right after startup. Startup and logins do not wait for it, graphs that were not in memory stay dormant, and storage.max_loaded_knowledge_graphs still bounds what is loaded. * fix(storage): drop graphs the warm-up leaves dormant at the loaded limit from the resident record A graph skipped because storage.max_loaded_knowledge_graphs was reached stayed recorded as in memory, so the next restart could warm stale names instead of the graphs that were loaded. * no-mistakes(review): fix(storage): stop warm-up loads unloading other knowledge graphs * no-mistakes(document): docs: note warm-up limit skip rewrites resident record
Intent
I want you to really hyper focus on making the symbolic engine, you know, crisp and really truly production ready. Like I really need you to, you know, put insane amount of effort into doing that.
(Context: plan ~/firstmate/data/nsdl-plan/report.md §6.1, GitHub milestone 7. Item EN-13: fast restart and measured restart time (#135 lazy loading, O(shards) startup) — why: RTO, cell rescheduling, and the pooled test engine (#135 is needed in M2))
Issue #135, "Lazy KG loading with LRU eviction; O(shards) startup": load_all_knowledge_graphs loads every KG eagerly and scans all shards once per KG (O(KGs x shards)), with JSON WAL replay. With one KG per user, startup time and RSS grow with the total data of all users. Group shards by KG once, load on first access, evict idle KGs, and load in parallel. Roadmap checkpoint: shard grouping already exists, but src/naming.rs scans known KG names per shard; measure and replace that ownership lookup rather than only parallelizing eager loading. Implement lazy activation, lifetime pinning for active requests/subscriptions, bounded safe eviction and metadata-only startup. Avoid a per-request LRU mutex; pass warm-query/delta and cold-start/RSS gates.
What Changed
storage_engine/residency.rsloads a graph on its first use: it reads and consolidates the graph's shards in parallel and applies the graph's WAL catalog changes. Concurrent first uses wait on a single load. Lookups on the hot path take no lock.storage.max_loaded_knowledge_graphsunloads the least recently used graphs once the limit is passed.storage.unload_idle_after_secsdrives a background task in the HTTP server that unloads graphs left idle that long. A graph stays loaded while any of these use it: a request handle, a held snapshot, a subscription pin, session schemas, the incremental engine, vector indexes, or catalog changes whose save failed. A reload publishes the same state under the same revision, and a writer that held a retired handle retries on the reloaded copy./metricsadds three new metrics:inputlayer_knowledge_graphs_loaded,inputlayer_knowledge_graph_loads_totalandinputlayer_knowledge_graph_unloads_total..backupexports now capture each graph at its own revision. Graphs in memory are still captured together under a brief commit hold. A graph that isn't loaded is blocked from loading only while its revision and catalogs are read, and its facts are read from disk afterwards. The manifest'srevisionis now the newest revision in the export. Also adds arestart_benchmarksbench, which measures open time, first-query time and RSS after a clean shutdown and after a crash with a WAL tail. Adds durability and residency tests, and updates the config, persistence, backup, REST API and architecture docs.Risk Assessment
✅ Low: The round-2 fix is small and closes the relation-drop gap.
drop_relationsetscatalog_unsavedwhen its schema or rule save fails.scrub_dropped_relationwrites both whole catalog files (a failedRuleCatalog::savekeepsdirtyset, so a later save retries) and sets the flag from that result. No other in-engine path changes a catalog and saves it outsidesave_catalogs,replay_catalog,drop_relationorscrub_dropped_relation, soreloadable()now covers every way a catalog can be left unsaved.Testing
I built the server, client and backup binaries at HEAD and ran a throwaway server (its own data dir, port and config) through nine live scenarios: metadata-only restart, lazy activation, the LRU cap, the idle timer with a subscription keeping its graph loaded, backup/verify/restore with dormant graphs, two failed-save cases (rule save and relation-drop save) made by removing write permission, a timed restart with 205 graphs, and a kill -9 restart. All nine passed and are recorded as CLI and metrics transcripts in the evidence directory. I also ran the targeted residency, online-checkpoint and durability tests, and all passed. Those tests cover the two fixes I couldn't trigger reliably through the server: the deadlock when one parallel query list names the same dormant graph twice, and a request to a dormant graph during a long backup. Because they were not driven live, that scenario is marked untested. One note, not caused by this change: shutdown with many graphs holding unflushed writes is slow, as described in the finding. There is no UI change, so I took no screenshots. I deleted the data directory afterwards.
near, al…Evidence: Restart reads metadata only (5 graphs on disk, 1 loaded)
Evidence: First use loads a dormant graph with the same facts and rules
Evidence: max_loaded_knowledge_graphs=2 LRU unloading
Evidence: Idle unload; subscription keeps its graph loaded and delivers a delta
Evidence: Online backup with dormant graphs, then verify, restore, query
Evidence: Failed rule-catalog save keeps the graph loaded
Evidence: Failed relation-drop save keeps the graph loaded; dropped schema stays gone
Evidence: Measured restart: 205 graphs / 400k facts ready in 81 ms, 48 MB RSS
Evidence: kill -9 then restart keeps committed fact and rule
Evidence: Residency test run (17 passed)
Evidence: Online checkpoint test run (8 passed)
Evidence: Node WebSocket subscription client used for scenario 4
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 3 issues found → auto-fixed (2) ✅
src/storage_engine/residency.rs:249-KnowledgeGraph::reloadable()does not check whether the in-memory rule and schema catalogs have been saved to the catalog files. When a durable catalog commit fails to save (install_cataloglogscatalog_save_failed_wal_keeps_changes, catalog_change.rs:364, program_commit.rs:120), memory holds the new rules but the files do not. The WAL keeps the records, but onlytake_recovered_catalogreads them, and that runs once at startup. Thentry_unload→to_dormantbuildsDormant { catalog: Vec::new(), last_published: Some((R, changes)) }(residency.rs:269-273). On reload,from_diskreads the stale catalog files and publishes them under the same revision R with the same change log. Result: a rule the client committed disappears in a running server, andexpect_revisionpreconditions pass against different state. This breaks the module's promise that a reload 'publishes the same state under the same snapshot revision'. Activation hits the same invariant at residency.rs:503: thecatalog_replay_save_failedpath keeps the changes in memory only, so a later unload drops them. Fix: track acatalog_unsavedflag onKnowledgeGraph. Set it whensave_catalogs/replay_catalogfails, clear it when a later save succeeds, and include it inreloadable().src/storage_engine/residency.rs:432-activateholds the slot's non-reentrantparking_lot::Mutexwhileload_graphrunsshards.par_iter()(residency.rs:458). The publicexecute_parallel_queries_on_knowledge_graphs(mod.rs:1958-1962) now callskg_handle(which may activate) from insidequeries.par_iter(). If a query list names the same dormant KG twice (e.g. after a restart), rayon worker A runs job 1, takes slot X's mutex and waits on its nested par_iter. While waiting, A work-steals job 2, which callsactivate(X)and blocks on the mutex A already holds. That deadlocks permanently. Before this change, lookups were a plain map read, so this could not happen. Smallest fix: inexecute_parallel_queries_on_knowledge_graphs, resolve every distinct KG's handle/snapshot sequentially before thepar_iter, or avoid holding the slot mutex across rayon joins inactivate.src/storage_engine/checkpoint.rs:124-capture_checkpointholds the slot mutex of every dormant KG at once. It then reads them from disk one by one (checkpoint.rs:142-148), releasing each only after its own full load. With one KG per user and most KGs dormant (the target workload), a.backupmakes the first request to any not-yet-read dormant KG wait until every KG sorted before it has been fully loaded from disk. That wait is O(total data), and those KGs can't be unloaded meanwhile. Before this change, a checkpoint held off commits only for O(KGs + relations) reference cloning. This is the author's deliberate consistency design, but it is an availability regression during backups that matters for the RTO/rescheduling goal. The remedy needs a design decision, not a mechanical repair, so it is listed for review. Options: read dormant shards filtered totime <= revisionwithout holding slots, or take a short hold per KG.🔧 Fix applied.
1 warning still open:
src/storage_engine/mod.rs:2500- This is a sibling the round-1 fix (unsaved-catalog-lost-on-unload) left behind. The fix setscatalog_unsavedonly insave_catalogs(catalog_change.rs:366) and the activation replay (residency.rs:498). The relation-drop paths also change the catalogs in memory and then save them, but a failed save there never sets the flag:KnowledgeGraph::drop_relation(mod.rs:2499-2504):remove_schemaremoves the schema from memory beforesave_schema_catalog(mod.rs:2692), andrule_catalog.dropremoves the rule beforesave()(rule_catalog.rs:697-702).scrub_dropped_relation(mod.rs:2531-2537).Failing sequence:
bhas a registered schematypedand facts intyped.drop_relation_in("b","typed")hits a catalog-file write error. It logsrelation_drop_cleanup_deferredand keeps the tombstone. Memory has no schema;schemas.jsonstill has it.bgoes idle.reloadable()is true, sotry_unloadunloads it.b.load_graphfilters the tombstoned shard but does not scrub catalogs; only startup (mod.rs:1862-1866) andsettle_relation_dropbefore a write to that name do.from_diskreloadstyped's schema (or a dropped rule) from the stale files.Result: in a running server, the dropped relation's schema or rule comes back (
has_schema_in/list_schemas, and rules depending on it) under the same published revision. This is the same 'reload publishes the same state' invariant the round-1 fix targeted.Fix: in
drop_relationandscrub_dropped_relation, setself.catalog_unsaved = truewhen the schema or rule save fails. Let a successfulscrub_dropped_relation(it saves both full catalogs) clear it. Alternatively, havereloadable()refuse while a relation tombstone for this KG exists.🔧 Fix applied.
✅ Re-checked - no issues remain.
src/storage_engine/mod.rs:992- Not caused by this change, but it affects restart time.StorageEngine::save_all, called at shutdown, flushes shards one by one withpersist.flush, and each flush rewrites the WAL. With 205 graphs written but not yet flushed, the debug build wrote about one shard per second, so shutdown took about 4 minutes. The 30 s graceful-shutdown timeout logged a warning, but the process stayed alive and kept holding the data-dir lock, so a new server could not start on the same directory. The base commit has the samesave_allloop. This branch's newflush_dirty(one WAL rewrite for all dirty shards) could probably be reused there.near, al…cargo build --bin inputlayer-server --bin inputlayer-client --bin inputlayer-backupat HEADLive: seeded alice/bob/carol over WebSocket (inputlayer-client --script), SIGTERM restart, read /metrics/prometheus and the kg_register_complete/kg_activated log linesLive: first query to dormantbobafter restart (recursive connected/2 plus a fact inserted before restart)Live: restart with max_loaded_knowledge_graphs=2, queried alice→bob→carol→alice, tracked loaded/loads/unloads metrics and kg_unloaded linesLive: unload_idle_after_secs=2 plus a Node WS client holding.subscribe pin1 ?connected(1, Y)on carol for 10 s, with an insert at t=6 s; per-second metricsLive:.backup snap1with all user graphs dormant,.backup status,inputlayer-backup verify,inputlayer-backup restore, second server on the restored dir, queried all graphsLive (adversarial): chmod a-w data/alice/rules, registered rulefar, waited past the idle timeout; then restored permissions, added rulenear, confirmed alice unloads and reloads with both rulesLive (adversarial): chmod a-w data/bob,.rel drop typedon a typed relation (relation_drop_cleanup_deferred), waited past the idle timeout, checked .rel; restarted and checked againLive: 200 extra graphs × 2000 facts, timed process start → /ready, RSS, first-query activation of dormant u150, warm repeatLive: insert plus rule into u10, kill -9, restart, query u10cargo test --lib storage_engine::residency(17 tests, including parallel_queries_naming_a_dormant_graph_twice_load_it_once and a_request_to_a_dormant_graph_is_served_during_a_backup)cargo test --test online_checkpoint_testscargo test --lib storage::persist::durability_tests✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.