Repository navigation
Conversation
…ed or moved (#93) An open WAL file outlives its path: when the data directory was removed or moved while the server ran, appends and fsyncs to the open file kept succeeding and inserts were acknowledged, but restart recovery reads wal/current.wal and never sees them. After each append the WAL now checks that the file it wrote is still the one at its path (device and inode). If not, the commit returns OutcomeUnknown (the record sits in the detached file, recovered only if it is put back) and the store refuses every later write with StoreReadOnly until restart. The directory is not recreated, since writes acknowledged earlier may have been lost with it.
This was referenced Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Intent
please continue developing the input layer engine so we make sure that the symbolic engine is ready to go live and completely production ready while we develop the GoTM uh, market.
(Context: the captain approved the neurosymbolic decision engine plan; its milestone 'Engine readiness' makes the symbolic engine production-ready for the hosted decision API. This task is item EN-1: #93: fact inserts report success when the WAL write fails (data directory gone) — why: a SaaS cannot acknowledge writes it did not store; P0 data safety)
What Changed
PersistWalnow records the device and inode ofwal/current.wal(on non-Unix platforms only a missing file is detected). It checks that the file at that path is still the same one after every append, and before it reopens the writer, reads the file, or rewrites it. If the file was removed, moved or replaced, the commit returnsOutcomeUnknowninstead of success. The WAL then refuses every later write withStoreReadOnlyuntil restart, and it does not quietly create a new, empty WAL. Truncation and rotation update the recorded file identity.wal_tests.rsand a new integration test,tests/data_dir_loss_tests.rs, cover two cases. First, inserts made after the data directory is removed returnoutcome_unknownand thenstore_read_only, and those inserts do not show up in the live store. Second, a write into a data directory that was moved away is recovered once the directory is put back.docs/guides/persistence.mdand the docs-site.mdxcopy) now explains what happens if the WAL file or data directory is removed or moved while the server is running.Risk Assessment
✅ Low: The fix round makes
file_idan Option that persists while the writer is closed. It is set at open from the recovered file, refreshed on reopen and after retain_ops' rename, and cleared only in clear(). check_in_place now runs before ensure_writer reopens the file and in settle(), so read_all, retain_ops, remove_shard_entries, clear and Drop all refuse a removed or replaced WAL with OutcomeUnknown/read-only instead of creating a fresh file. The open-writer append path still does only one stat. The new tests drive the public WAL API and check observable refusals and file state.Testing
I built the server and client from this change and drove five live scenarios on a disposable data directory. They covered removing the data dir, moving it away and restoring it across a restart, deleting
current.walwhile the writer was closed after a flush (forced with BUFFER_SIZE=2), and replacing the WAL with a different file with the writer open and closed. A happy path with many WAL self-rewrites followed by a restart also passed. In every failure case the insert returnedoutcome unknownrather than success, and later writes were refused as read-only. No empty WAL was created, and the refused rows never appeared in queries. A moved directory that was put back recovered the uncertain write, and writes resumed after restart. The change's targeted integration and WAL unit tests also pass. During setup I briefly thought SIGTERM shutdown hung, but my harness had signalled the wrong pid; with correct process tracking every server exited in about 2s. I removed all temp dirs and servers, and the worktree is unchanged.Evidence: Data dir removed while running: insert refused (outcome unknown), then read-only
Evidence: Data dir moved then restored: uncertain write recovered after restart, writes resume
Evidence: WAL file deleted while writer closed after flush: refused, no fresh WAL created
Evidence: Happy path across WAL self-rewrites + WAL replaced (writer open / closed)
Evidence: Targeted test output
Pipeline
Updates from git push no-mistakes
✅ **intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
🔧 **Review** - 1 issue found → auto-fixed ✅
src/storage/persist/wal.rs:183- The in-place check only covers a WAL file that stays open. Several paths close the writer and later reopen it withOpenOptions::create(true):settle()(viaread_all,retain_ops,remove_shard_entries, which runs on every buffer-full flush at persist/mod.rs:796) anddiscard_writer. Becauseensure_writerre-readsfile_idfrom whatever is at the path when it reopens, a file removed or replaced while the writer is closed goes unnoticed. Example: a commit fills shard A's buffer, soflush_existing(A)callsremove_shard_entries, which leaveswriter = Nonewhilecurrent.walstill holds acknowledged records for unflushed shard B. Ifwal/current.walis then deleted (directory left intact), the next append creates a fresh empty file, the check passes, and the write is acknowledged. Restart then silently loses B's earlier acknowledged writes. Replacing the file in that window has the same effect. Removing or moving the whole data directory, the issue's main scenario, is still caught here, because the reopen fails on the missing parent directory and nothing is acknowledged. Suggested fix: keep an in-memory expectation of whethercurrent.walshould exist and which file it is (record the identity aftersettle/retain_ops' rename, clear it inclear()). Inensure_writer, if the expected file is missing or has a different device/inode, take the same read-only/OutcomeUnknown path instead of creating a new file.🔧 Fix applied.
✅ Re-checked - no issues remain.
✅ **Test** - passed
✅ No issues found.
Builttarget/debug/inputlayer-serverandinputlayer-clientfrom the change and started the server on a throwaway data dir in a mktemp directory (port 18093, bootstrap API key via env), driven over WebSocket withinputlayer-clientScenario 1: inserted+claim[("c1", 1)], ranrm -rfon the data dir with the server running, then inserted c2 and c3 and added a rule; then queried?claim(C, V)Scenario 2: inserted c1,mv data movedwith the server running, inserted c2, stopped the server with SIGTERM, ranmv moved data, restarted, queried, inserted c3Scenario 3: started the server withINPUTLAYER_STORAGE__PERSIST__BUFFER_SIZE=2, inserted intoother, filledclaim's buffer so the flush rewrote the WAL (inode changed) and closed the writer, ranrm current.walwith the directory left intact, then inserted againScenario 4A: with BUFFER_SIZE=2, ran 14 alternating inserts across two relations (many WAL self-rewrites), restarted, verified 7+7 rows and that a new insert succeededScenario 4B/4C: replacedcurrent.walwith a byte-identical copy (cp+mv, new inode), once with the writer open and once with it closed after a flush, then insertedcargo test --test data_dir_loss_testscargo test --lib storage::persist::waldocs/guides/persistence.md:160- The persistence guide exists in two full copies, docs/guides/persistence.md and docs/content/docs/guides/persistence.mdx. This change updated both the same way, but the copies already disagree elsewhere: the .md copy documentscompaction_windowwhere the .mdx documentsauto_compact_threshold/auto_compact_interval_secs. Follow-up: make one of them the owner and reduce the other to a pointer or generate it.✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.
Deferred heavy checks
Per the dev-box rule, only the checks for the touched area ran here: fmt, clippy, all lib tests (3030), the WAL unit tests, the new engine-level
tests/data_dir_loss_tests.rs, and the persistence integration suites (persist_crash,persist_roundtrip,crash_recovery_tests,atomic_program_tests,restart_equivalence,data_dir_lock_tests). An end-to-end WS repro against a release build confirmedoutcome_unknownthenstore_read_onlyafterrm -rformvof the data dir. Themake perf-gatesame-host performance gate is deferred to the benchmark machine at the release checkpoint. The hot-path cost is onestatper WAL append and one per flush rewrite, with no new locks.