Repository navigation
Conversation
There was no supported way to back up a data directory: copying it while a server runs gives an inconsistent WAL/shard set, and nothing checked a copy before it was restored. inputlayer-backup (shipped in the engine image) adds three commands: - create: takes the S01 data_dir LOCK for the whole copy, so it refuses a directory a running server holds and no server can start mid-copy. Copies metadata, WAL, shards/batches, per-KG catalogs and the _internal auth KG (users, key hashes, ACLs), fsyncs everything, and writes a manifest (every directory, every file's size and SHA-256) last. - verify: read-only check that the backup holds exactly the manifest's entries with matching sizes and digests. - restore: checks every file against the manifest as it copies, holds the target's lock meanwhile, then opens the result with StorageEngine as server startup does and prints a per-KG summary. Every command writes only into a new or empty directory (a volume's lost+found is allowed) and never overwrites data; a destination inside the source is refused before anything is created. A failed run returns the destination to how it found it; a killed one leaves no manifest and is refused as incomplete. Manifest paths cannot escape the target. Off the hot path: a new library module and binary; the server's code paths are unchanged. Measured on an 11 MB / 550k-fact directory: create 31 ms, verify 78 ms, restore 28 ms (files are copied in parallel so fsyncs overlap; sequential copying measured 6-60x slower on this host). Docs: docs/guides/backup.md (+ site page) with systemd, Docker Compose and Kubernetes procedures. Partial #48 (online checkpoint remains).
This was referenced Oct 3, 2026
Contributor
Author
|
Superseded by #216 (merged), which carried this R06 commit. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft for visibility. Stacked on the release checkpoint 1 branch (
fm/il-rc1); it will be retargeted tomainonce checkpoint 1 lands, and validated through the no-mistakes pipeline before merge.Tracker: #199