Skip to content

ops: restore-tested offline backup tool and runbook (#48) - #201

Closed
jsam wants to merge 1 commit into
fm/il-rc1from
fm/il-r06
Closed

jsam wants to merge 1 commit into
fm/il-rc1from
fm/il-r06

Conversation

@jsam

@jsam jsam commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Draft for visibility. Stacked on the release checkpoint 1 branch (fm/il-rc1); it will be retargeted to main once checkpoint 1 lands, and validated through the no-mistakes pipeline before merge.

Tracker: #199

There was no supported way to back up a data directory: copying it while
a server runs gives an inconsistent WAL/shard set, and nothing checked a
copy before it was restored.

inputlayer-backup (shipped in the engine image) adds three commands:

- create: takes the S01 data_dir LOCK for the whole copy, so it refuses a
  directory a running server holds and no server can start mid-copy.
  Copies metadata, WAL, shards/batches, per-KG catalogs and the _internal
  auth KG (users, key hashes, ACLs), fsyncs everything, and writes a
  manifest (every directory, every file's size and SHA-256) last.
- verify: read-only check that the backup holds exactly the manifest's
  entries with matching sizes and digests.
- restore: checks every file against the manifest as it copies, holds
  the target's lock meanwhile, then opens the result with StorageEngine
  as server startup does and prints a per-KG summary.

Every command writes only into a new or empty directory (a volume's
lost+found is allowed) and never overwrites data; a destination inside
the source is refused before anything is created. A failed run returns
the destination to how it found it; a killed one leaves no manifest and
is refused as incomplete. Manifest paths cannot escape the target.

Off the hot path: a new library module and binary; the server's code
paths are unchanged. Measured on an 11 MB / 550k-fact directory: create
31 ms, verify 78 ms, restore 28 ms (files are copied in parallel so
fsyncs overlap; sequential copying measured 6-60x slower on this host).

Docs: docs/guides/backup.md (+ site page) with systemd, Docker Compose
and Kubernetes procedures.

Partial #48 (online checkpoint remains).
@jsam

jsam commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #216 (merged), which carried this R06 commit.

@jsam jsam closed this Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant