Repository navigation
Conversation
There was no supported way to back up a data directory: copying it while a server runs gives an inconsistent WAL/shard set, and nothing checked a copy before it was restored. inputlayer-backup (shipped in the engine image) adds three commands: - create: takes the S01 data_dir LOCK for the whole copy, so it refuses a directory a running server holds and no server can start mid-copy. Copies metadata, WAL, shards/batches, per-KG catalogs and the _internal auth KG (users, key hashes, ACLs), fsyncs everything, and writes a manifest (every directory, every file's size and SHA-256) last. - verify: read-only check that the backup holds exactly the manifest's entries with matching sizes and digests. - restore: checks every file against the manifest as it copies, holds the target's lock meanwhile, then opens the result with StorageEngine as server startup does and prints a per-KG summary. Every command writes only into a new or empty directory (a volume's lost+found is allowed) and never overwrites data; a destination inside the source is refused before anything is created. A failed run returns the destination to how it found it; a killed one leaves no manifest and is refused as incomplete. Manifest paths cannot escape the target. Off the hot path: a new library module and binary; the server's code paths are unchanged. Measured on an 11 MB / 550k-fact directory: create 31 ms, verify 78 ms, restore 28 ms (files are copied in parallel so fsyncs overlap; sequential copying measured 6-60x slower on this host). Docs: docs/guides/backup.md (+ site page) with systemd, Docker Compose and Kubernetes procedures. Partial #48 (online checkpoint remains).
A running server could not be backed up: the offline tool (R06) refuses a live data directory, and copying one while it serves gives an inconsistent WAL/shard set. `.backup [name]` (admin only) now exports a checkpoint of every knowledge graph at one committed revision while the server keeps serving: - Capture holds every KG's read lock at once and reads the logical clock. Every revision is assigned and applied under its KG's write lock, so no commit is in flight: the export holds exactly the commits up to the named revision. KG creation waits for the capture (new `kg_set` gate, taken only by creation and capture). Only shared references are taken, so commits are held for ~0.1 ms idle / ~1.2 ms under load on 51 KGs, 1,001 relations, 501k facts; queries read snapshots and are not held. - One background thread writes the checkpoint as a compacted data directory (one batch per relation, no WAL, rule/schema/index catalogs, KG listing), then syncs and hashes every file on four dedicated threads and writes the R06 manifest last, now recording the revision. `inputlayer-backup verify` and `restore` accept it unchanged and print the revision. - The destination is claimed before capture: a non-empty directory, a backup_dir inside the data directory or a non-plain name is refused at once. One export at a time; `.backup status` reports progress or outcome. A failed or cancelled export removes what it wrote, and dropping the engine cancels a running export and waits for that cleanup; a killed one leaves no manifest and is refused as incomplete. Config: `storage.backup_dir` (unset disables `.backup`). Measured (release, same host): commit p99 ~0.26 ms and query p99 within noise with exports running back to back; export write 1.5-1.7 s for 501k facts. Writing through FilePersist's per-shard flush took 27.9 s (fsync-bound), hence `persist::ExportWriter`, which writes batches and shard metadata without per-file syncs; the parquet encoder and shard-meta JSON were split out of their durable writers for it, with no behavior change there. Tests: a cut under concurrent cross-KG writes restores exactly `revision - base` commits (a per-KG capture fails it); `.backup` through the handler restores the served queries, rules, users, ACLs and a vector index and is denied to a viewer; cancel, shutdown mid-export, missing manifest, flipped byte, unsafe destinations, and capture liveness under KG churn. Closes #48
…log items make doc-check (rustdoc -D warnings) rejects intra-doc links from public docs to private items; name them in plain text instead.
jsam
marked this pull request as ready for review
October 3, 2026 20:29
jsam
added a commit
that referenced
this pull request
Oct 4, 2026
…ry request (#249) The release-checkpoint performance gate fails on main: cheap_query serial p50 is 1.110x the approved baseline e732cd5 on the 32-vCPU runner (115 -> 126 us). Concurrent latency and throughput did not move. Bisected to #214 (per-connection request pipelining), made worse by #222. Cause: the pipeline runs requests in a `FuturesUnordered`. Its `poll_next` wakes its own task and returns `Pending` after any poll that polled every future in the set. With one request in flight, which is every serial client, that happens on each request's first poll. The connection task wakes itself, tokio schedules that as a yield, and a yield wakes an idle worker to take the task. Each request therefore costs one more context switch (4.0 -> 5.0 per query) and a hand-off of the connection task to another core. #222 added no work per query but made the hand-off dearer: user cycles per query rose 15% at the same instruction count. Fix: keep one never-woken future in the set. After the pipeline's first poll it is never polled again, so no poll reaches every future and the forced yield no longer fires. The yield after futures that wake themselves still applies. Pipelining, ordering, deadlines, cancellation and the credential fence are unchanged. Nothing is added per request: no lock, atomic or allocation. Each connection gets one more set node. Bisect (cheap_query, `?edge(1, Y)`, serial). Server counters per query from perf stat. Ratios are against e732cd5, from an interleaved probe with one pinned server per commit, 40 rounds: | commit | PR | user instr/query | ctx switches/query | p50 ratio | |---|---|---|---|---| | e732cd5 | baseline | 1.282M | 4.0 | 1.000 | | d75ce6e | #208 atomic programs | 1.279M | 4.0 | 0.993 | | 4578fe6 | #215 notification stream | 1.271M | 4.0 | 0.991 | | 71a4d24 | #216 online backup | 1.275M | 4.0 | 1.016 | | 2f621c5 | #200 first-update latency | 1.272M | 4.0 | 1.009 | | e96df5c | #207 | 1.279M | 4.0 | 0.998 | | 4dd8fe9 | #209 shard index | 1.272M | 4.0 | 0.987 | | 277c7f0 | #206 | 1.274M | 4.0 | 1.008 | | 3554c98 | #214 pipelining, deadlines | 1.296M | 5.0 | 1.020 | | 3492a59 | #213 | 1.294M | 5.0 | 1.030 | | 2ba9a5a | #212 provenance | 1.297M | 5.0 | 1.029 | | 9ed61b4 | #204 key expiry | 1.297M | 5.0 | 1.027 | | be35ee3 | #221 | 1.297M | 5.0 | 1.013 | | c98a590 | #222 bounded streaming | 1.298M | 5.0 | 1.047 | | a6fd628 | #219 wildcard fix | 1.297M | 5.0 | 1.061 | | b7f7ad6 | #220 WAL durability | 1.300M | 5.0 | 1.056 | | de54807 | #223 shared standing eval | 1.299M | 5.0 | 1.071 | | 55ec34d | #225 checkpoint-1 fixes | 1.300M | 5.0 | 1.045 | | 23712b8 | main | 1.300M | 5.0 | 1.064 | | this change | | 1.300M | 4.0 | 1.033 | Confirming A/Bs with the gate's own fixture (fresh pinned servers each round, 40 rounds, ratio [95% interval]): - #214 vs its parent 277c7f0: 1.039 [1.027..1.049] - #222 vs its parent be35ee3: 1.025 [1.006..1.044] - A/A 3554c98: 1.010 [1.002..1.025] - e732cd5 -> main: 1.048 [1.028..1.066] - main -> this change: 0.978 [0.958..0.989] - e732cd5 -> this change: 1.021 [1.009..1.044], PASS (budget 1.05) User cycles per query (perf stat, same instruction count): e732cd5 958K, be35ee3 1053K, c98a590 1206K, this change 1006K. Full gate, all fixtures, 20 rounds, servers pinned to CPUs 2-15: - main -> this change: no metric worse. Every required metric is PASS, or INCONCLUSIVE with an interval that includes 1.0. cheap_query p50 0.979 [0.953..1.007]. - e732cd5 -> this change: cheap_query p50 1.029 [1.019..1.072], INCONCLUSIVE on a busy host (load average up to 22); bound_query p50 1.022, PASS. The insert and delta metrics improved since the baseline. The shared host is about 4x slower per query than the CI runner, so the same hand-off is a smaller share of each query here than there. The remaining ~1.7% of user instructions per query since #214 is the pipeline's own per-request bookkeeping: the boxed request future (one 4-8 KiB allocation), its set node, the request control and the in-flight maps.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
R07: consistent online checkpoint/export while the engine runs (.backup), on top of T01-T03 (#208, on main). Includes R06 (#201, the offline backup tool) as its first commit, so #201 is superseded once this merges. Also restores plain-text names for private items in rustdoc (main's doc-check regressed after #215). Validated through no-mistakes before merge.
Tracker: #199