fix(storage): fail catalog saves when the directory fsync fails so committed rules stay in the WAL by jsam · Pull Request #333 · inputlayer/inputlayer

jsam · 2026-10-05T12:38:58Z

Intent

I want you to really hyper focus on making the symbolic engine, you know, crisp and really truly production ready. (Context: #297 (audit P1-1): the catalog save discards the parent-directory fsync error, so a committed rule or schema can vanish on power loss. save_json_atomic ignores the directory fsync error and returns Ok; for catalog-only commits the WAL entries are then pruned away, so a power loss can silently drop a committed rule/schema. Fix: propagate the directory fsync error like sync_directory does; add a fault-injection test.)

What Changed

save_json_atomic (src/storage/metadata.rs) no longer ignores errors from opening or fsyncing the parent directory after the rename. It now calls the shared persist::sync_directory (now pub(crate)) and returns its error. A path with no parent or an empty parent falls back to .. The result is that a catalog or schema save whose rename may not be durable counts as a failed save, and its WAL entries are not pruned.
Adds fault-injection tests:
- A metadata unit test checks that save returns an error when the directory sync fails.
- A catalog-commit test injects a sync fault on the rules/ directory and on the KG directory, then simulates power loss by deleting rules/catalog.json and schema.json. It checks that the committed schema and rule are still in the WAL and come back on reopen.

Risk Assessment

✅ Low: The change is small and targeted. save_json_atomic now returns the directory-fsync error through the existing sync_directory helper, and every caller already handles a save error safely: save_catalogs leaves the WAL entries in place, the rule catalog stays dirty, and tombstones are left unchanged in memory. Fault-injection tests cover both the rules directory and the KG directory.

Testing

I ran the two new fault-injection tests and the existing WAL-prune test. All three pass, and the two new tests fail with the pre-fix save_json_atomic. Then I drove the real inputlayer-server and client end to end, using an LD_PRELOAD shim to make directory fsync fail with EIO. With the fix, the commit succeeds and logs catalog_save_failed_wal_keeps_changes, and the WAL keeps the catalog entries. After a kill -9 and deleting the non-durable catalog files, a restart replays rule v and the item int schema: ?v(X) returns 1 and 2, and a string insert is rejected. The pre-fix control logs no warning, prunes the WAL, and comes back with no rules and ?v(X) returning nothing. All four live scenarios passed. The unit-test regression scenario is marked untested in the live sense because it ran only as cargo tests. The target/ build output was removed afterwards.

Live validation: ✅ go - 4 of 5 scenarios driven live against the product

Scenario	Result	Live	Evidence
A failing directory fsync during a catalog save is reported as a failed save, not Ok	✅ pass	live	faulty.log: '[failfsync] injected EIO on fsync(dir=.../default/rules)' then 'WARN catalog_save_failed_wal_keeps_changes ... Failed to write catalog: I/O error: Input/output error (os error 5)', and th…
A catalog-only commit (schema + rule) whose directory fsync failed keeps its changes in the WAL instead of pruning them	✅ pass	live	scenario.log: current.wal is 449 bytes and mentions item after the commit; scenario-base.log (pre-fix): the WAL directory is empty after the same commit
After a power loss drops the un-synced catalog renames, a restarted server still has the committed rule and schema	✅ pass	live	scenario.log: after kill -9 and deleting rules/catalog.json and schema.json, the restart shows 'v (1 clause(s))', ?v(X) returns 1 and 2, the string insert into item is rejected, and the catalog files…
Regression tests fail before the fix and pass after it (fault injection on both the rules dir and the KG dir)	⏸️ untested	no	The prior payload established this only through cargo unit tests using cfg(test) inject_sync_fault, not against the live product, so no live result was recorded for this scenario. The live behaviour i…
Healthy path: when directory fsync succeeds, saved catalog changes still leave the WAL	✅ pass	live	scenario.log: after the fault-free restart rewrote the catalogs, current.wal shrank to 128 bytes; the saved_catalog_changes_leave_the_wal test passes

Evidence: Live transcript, fixed server: fault-injected commit, WAL retained, power-loss simulation, recovery of rule and schema

== WAL before ==
total 8
drwxrwxr-x 2 sam sam 4096 Oct  5 12:34 .
drwxrwxr-x 5 sam sam 4096 Oct  5 12:34 ..
== client: commit schema + rule while dir fsync fails ==
Connecting to server at http://127.0.0.1:18471...
Connected!

Server status: healthy
Authenticated as: admin
Current knowledge graph: default

> .kg use default
Switched to knowledge graph: default
> +item(x: int)
Schema for 'item' registered with 1 columns (persistent)
> +v(X) <- item(X)
Rule 'v' registered.
> .rule list
Rules:
  v (1 clause(s))
exit=0
== WAL after faulty commit (catalog entries retained) ==
total 12
drwxrwxr-x 2 sam sam 4096 Oct  5 12:34 .
drwxrwxr-x 5 sam sam 4096 Oct  5 12:34 ..
-rw-rw-r-- 1 sam sam  449 Oct  5 12:34 current.wal
item mentions: 2
== simulate power loss: kill -9, then drop catalog files whose renames were not durable ==
~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data/default/schema.json
~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data/default/rules/catalog.json
removed '~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data/default/rules/catalog.json'
removed '~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data/default/schema.json'
== client after restart (no fault) ==
Connecting to server at http://127.0.0.1:18471...
Connected!

Server status: healthy
Authenticated as: admin
Current knowledge graph: default

> .kg use default
Switched to knowledge graph: default
> .rule list
Rules:
  v (1 clause(s))
> .rel
No relations in current knowledge graph.
> +item[(1), (2)]
Inserted 2 fact(s) into 'item'.
> ?v(X)
┌───┐
│ X │
├───┤
│ 1 │
│ 2 │
└───┘
2 rows
exit=0
== catalog files rewritten after recovery ==
~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data/default/schema.json
~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data/default/rules/catalog.json
total 12
drwxrwxr-x 2 sam sam 4096 Oct  5 12:35 .
drwxrwxr-x 5 sam sam 4096 Oct  5 12:34 ..
-rw-rw-r-- 1 sam sam  128 Oct  5 12:35 current.wal
== schema check after recovery: string into int column must be rejected ==
Switched to knowledge graph: default
> +item[("not-an-int")]
Insert rejected for 'item': Insert rejected for 'item': batch of 1 tuples had type/arity errors

Evidence: Fixed server log showing injected EIO and catalog_save_failed_wal_keeps_changes warnings

InputLayer Server
=================

�[2m2026-10-05T12:34:42.334105Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m data_dir_lock_acquired �[3mdata_dir�[0m�[2m=�[0m/home/sam/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data �[3melapsed_us�[0m�[2m=�[0m110
�[2m2026-10-05T12:34:42.514191Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_load_complete �[3mknowledge_graphs�[0m�[2m=�[0m2 �[3melapsed_ms�[0m�[2m=�[0m2
�[2m2026-10-05T12:34:42.515305Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_memory_limits �[3mquery_memory_bytes�[0m�[2m=�[0m4294967296 �[3mtotal_query_memory_bytes�[0m�[2m=�[0m0 �[3mcompute_permits�[0m�[2m=�[0m24
�[2m2026-10-05T12:34:42.515335Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0m_internal �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:42.515368Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m Auth bootstrap: admin user already exists
Storage engine initialized

HTTP Server
-----------
Address: 127.0.0.1:18471

HTTP server listening on: http://127.0.0.1:18471
WebSocket API docs at: http://127.0.0.1:18471/api/ws-docs
�[2m2026-10-05T12:34:48.365321Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::rest::handlers::ws�[0m�[2m:�[0m ws_connection_start �[3mkg�[0m�[2m=�[0mdefault
�[2m2026-10-05T12:34:48.365568Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m audit_auth_apikey_success �[3musername�[0m�[2m=�[0m"admin" �[3mcredential�[0m�[2m=�[0mapikey:bootstrap#2
�[2m2026-10-05T12:34:48.365603Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::rest::handlers::ws�[0m�[2m:�[0m ws_authenticated �[3mkg�[0m�[2m=�[0mdefault �[3musername�[0m�[2m=�[0madmin �[3mrole�[0m�[2m=�[0madmin �[3mcredential�[0m�[2m=�[0mapikey:bootstrap#2
�[2m2026-10-05T12:34:48.365625Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mdefault �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.365678Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::rest::handlers::ws�[0m�[2m:�[0m ws_session_created �[3mkg�[0m�[2m=�[0mdefault �[3mactive_sessions�[0m�[2m=�[0m1
�[2m2026-10-05T12:34:48.367392Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_start �[3mprogram_len�[0m�[2m=�[0m15 �[3mprogram_preview�[0m�[2m=�[0m.kg use default �[2m�[3mrequest_id�[0m�[2m=�[0m1 �[3mmsg_bytes�[0m�[2m=�[0m55�[0m
�[2m2026-10-05T12:34:48.367664Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_start �[3mprogram_len�[0m�[2m=�[0m15
�[2m2026-10-05T12:34:48.367720Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mdefault �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.367765Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_parse_complete �[3mprogram_len�[0m�[2m=�[0m15 �[3mstatements�[0m�[2m=�[0m1 �[3mparse_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.367793Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m meta_kg_use_start �[3mkg�[0m�[2m=�[0mdefault
�[2m2026-10-05T12:34:48.367802Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mdefault �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.367810Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m meta_kg_use_ok �[3mkg�[0m�[2m=�[0mdefault
�[2m2026-10-05T12:34:48.367867Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_complete_messages �[3mprogram_len�[0m�[2m=�[0m15 �[3mtotal_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.368002Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_complete �[3mprogram_len�[0m�[2m=�[0m15 �[3mcompute_ms�[0m�[2m=�[0m0 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m1 �[3mmsg_bytes�[0m�[2m=�[0m55�[0m
�[2m2026-10-05T12:34:48.368051Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_end �[3mprogram_len�[0m�[2m=�[0m15 �[3melapsed_ms�[0m�[2m=�[0m0 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m1 �[3mmsg_bytes�[0m�[2m=�[0m55�[0m
�[2m2026-10-05T12:34:48.368771Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_start �[3mprogram_len�[0m�[2m=�[0m13 �[3mprogram_preview�[0m�[2m=�[0m+item(x: int) �[2m�[3mrequest_id�[0m�[2m=�[0m2 �[3mmsg_bytes�[0m�[2m=�[0m53�[0m
�[2m2026-10-05T12:34:48.369015Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_start �[3mprogram_len�[0m�[2m=�[0m13
�[2m2026-10-05T12:34:48.369040Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mdefault �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.369059Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_parse_complete �[3mprogram_len�[0m�[2m=�[0m13 �[3mstatements�[0m�[2m=�[0m1 �[3mparse_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.463674Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine::program_commit�[0m�[2m:�[0m program_commit_persisted �[3mkg�[0m�[2m=�[0mdefault �[3mrelations�[0m�[2m=�[0m0 �[3mstatements�[0m�[2m=�[0m1 �[3mtime�[0m�[2m=�[0m4 �[3mpersist_ms�[0m�[2m=�[0m94
[failfsync] injected EIO on fsync(dir=~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data/default/rules)
[failfsync] injected EIO on fsync(dir=~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data/default)
�[2m2026-10-05T12:34:48.540282Z�[0m �[33m WARN�[0m �[2minputlayer::storage_engine::catalog_change�[0m�[2m:�[0m catalog_save_failed_wal_keeps_changes �[3mkg�[0m�[2m=�[0mdefault �[3mrevision�[0m�[2m=�[0m4 �[3merror�[0m�[2m=�[0mFailed to write catalog: I/O error: Input/output error (os error 5)
�[2m2026-10-05T12:34:48.540319Z�[0m �[33m WARN�[0m �[2minputlayer::storage_engine::catalog_change�[0m�[2m:�[0m catalog_save_failed_wal_keeps_changes �[3mkg�[0m�[2m=�[0mdefault �[3mrevision�[0m�[2m=�[0m4 �[3merror�[0m�[2m=�[0mFailed to save schema catalog: IO error: Failed to write schema catalog: I/O error: Input/output error (os error 5)
�[2m2026-10-05T12:34:48.540406Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m snapshot_publish_complete �[3mrelations�[0m�[2m=�[0m0 �[3mrules�[0m�[2m=�[0m0 �[3msnapshot_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.540455Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_statement_exec_complete �[3mprogram_len�[0m�[2m=�[0m13 �[3mstmt_exec_ms�[0m�[2m=�[0m171 �[3msession_facts�[0m�[2m=�[0m0 �[3msession_rules�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.540472Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_complete_messages �[3mprogram_len�[0m�[2m=�[0m13 �[3mtotal_ms�[0m�[2m=�[0m171
�[2m2026-10-05T12:34:48.540773Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_complete �[3mprogram_len�[0m�[2m=�[0m13 �[3mcompute_ms�[0m�[2m=�[0m171 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m2 �[3mmsg_bytes�[0m�[2m=�[0m53�[0m
�[2m2026-10-05T12:34:48.540880Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_end �[3mprogram_len�[0m�[2m=�[0m13 �[3melapsed_ms�[0m�[2m=�[0m172 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m2 �[3mmsg_bytes�[0m�[2m=�[0m53�[0m
�[2m2026-10-05T12:34:48.541991Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_start �[3mprogram_len�[0m�[2m=�[0m16 �[3mprogram_preview�[0m�[2m=�[0m+v(X) <- item(X) �[2m�[3mrequest_id�[0m�[2m=�[0m3 �[3mmsg_bytes�[0m�[2m=�[0m56�[0m
�[2m2026-10-05T12:34:48.542283Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_start �[3mprogram_len�[0m�[2m=�[0m16
�[2m2026-10-05T12:34:48.542316Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mdefault �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.542354Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_parse_complete �[3mprogram_len�[0m�[2m=�[0m16 �[3mstatements�[0m�[2m=�[0m1 �[3mparse_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.597502Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine::program_commit�[0m�[2m:�[0m program_commit_persisted �[3mkg�[0m�[2m=�[0mdefault �[3mrelations�[0m�[2m=�[0m0 �[3mstatements�[0m�[2m=�[0m1 �[3mtime�[0m�[2m=�[0m5 �[3mpersist_ms�[0m�[2m=�[0m54
[failfsync] injected EIO on fsync(dir=~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data/default/rules)
[failfsync] injected EIO on fsync(dir=~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/run/data/default)
�[2m2026-10-05T12:34:48.670496Z�[0m �[33m WARN�[0m �[2minputlayer::storage_engine::catalog_change�[0m�[2m:�[0m catalog_save_failed_wal_keeps_changes �[3mkg�[0m�[2m=�[0mdefault �[3mrevision�[0m�[2m=�[0m5 �[3merror�[0m�[2m=�[0mFailed to write catalog: I/O error: Input/output error (os error 5)
�[2m2026-10-05T12:34:48.670532Z�[0m �[33m WARN�[0m �[2minputlayer::storage_engine::catalog_change�[0m�[2m:�[0m catalog_save_failed_wal_keeps_changes �[3mkg�[0m�[2m=�[0mdefault �[3mrevision�[0m�[2m=�[0m5 �[3merror�[0m�[2m=�[0mFailed to save schema catalog: IO error: Failed to write schema catalog: I/O error: Input/output error (os error 5)
�[2m2026-10-05T12:34:48.670787Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m snapshot_publish_complete �[3mrelations�[0m�[2m=�[0m0 �[3mrules�[0m�[2m=�[0m1 �[3msnapshot_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.670894Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_statement_exec_complete �[3mprogram_len�[0m�[2m=�[0m16 �[3mstmt_exec_ms�[0m�[2m=�[0m128 �[3msession_facts�[0m�[2m=�[0m0 �[3msession_rules�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.670929Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_complete_messages �[3mprogram_len�[0m�[2m=�[0m16 �[3mtotal_ms�[0m�[2m=�[0m128
�[2m2026-10-05T12:34:48.671130Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_complete �[3mprogram_len�[0m�[2m=�[0m16 �[3mcompute_ms�[0m�[2m=�[0m128 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m3 �[3mmsg_bytes�[0m�[2m=�[0m56�[0m
�[2m2026-10-05T12:34:48.671221Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_end �[3mprogram_len�[0m�[2m=�[0m16 �[3melapsed_ms�[0m�[2m=�[0m129 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m3 �[3mmsg_bytes�[0m�[2m=�[0m56�[0m
�[2m2026-10-05T12:34:48.672137Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_start �[3mprogram_len�[0m�[2m=�[0m10 �[3mprogram_preview�[0m�[2m=�[0m.rule list �[2m�[3mrequest_id�[0m�[2m=�[0m4 �[3mmsg_bytes�[0m�[2m=�[0m50�[0m
�[2m2026-10-05T12:34:48.672305Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_start �[3mprogram_len�[0m�[2m=�[0m10
�[2m2026-10-05T12:34:48.672335Z�[0m �[32m INFO�[0m �[2minputlayer::storage_engine�[0m�[2m:�[0m kg_ensure_exists �[3mkg�[0m�[2m=�[0mdefault �[3melapsed_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.672367Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_parse_complete �[3mprogram_len�[0m�[2m=�[0m10 �[3mstatements�[0m�[2m=�[0m1 �[3mparse_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.672401Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_job_complete_messages �[3mprogram_len�[0m�[2m=�[0m10 �[3mtotal_ms�[0m�[2m=�[0m0
�[2m2026-10-05T12:34:48.672503Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::handler�[0m�[2m:�[0m query_complete �[3mprogram_len�[0m�[2m=�[0m10 �[3mcompute_ms�[0m�[2m=�[0m0 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m4 �[3mmsg_bytes�[0m�[2m=�[0m50�[0m
�[2m2026-10-05T12:34:48.672536Z�[0m �[32m INFO�[0m �[1mws_request�[0m: �[2minputlayer::protocol::rest::handlers::ws::execute�[0m�[2m:�[0m ws_execute_end �[3mprogram_len�[0m�[2m=�[0m10 �[3melapsed_ms�[0m�[2m=�[0m0 �[3mok�[0m�[2m=�[0mtrue �[2m�[3mrequest_id�[0m�[2m=�[0m4 �[3mmsg_bytes�[0m�[2m=�[0m50�[0m
�[2m2026-10-05T12:34:48.673005Z�[0m �[32m INFO�[0m �[2minputlayer::protocol::rest::handlers::ws�[0m�[2m:�[0m ws_session_disconnecting �[3mactive_sessions�[0m�[2m=�[0m1

Evidence: Control transcript, pre-fix server: fsync error swallowed, WAL pruned, rule lost after power loss

== BASE (pre-fix) server: commit while dir fsync fails ==
> +v(X) <- item(X)
Rule 'v' registered.
> .rule list
Rules:
  v (1 clause(s))
[failfsync] injected EIO on fsync(dir=~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/base/data/default/rules)
[failfsync] injected EIO on fsync(dir=~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/base/data/default)
[failfsync] injected EIO on fsync(dir=~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/base/data/default/rules)
[failfsync] injected EIO on fsync(dir=~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/base/data/default)
-- WAL:
drwxrwxr-x 5 sam sam 4096 Oct  5 12:35 ..
/bin/bash: line 16: 216550 Killed                  FAIL_FSYNC_DIRS=$D/default/rules:$D/default LD_PRELOAD=$E/failfsync.so INPUTLAYER_BOOTSTRAP_API_KEY=$K $S --port 18472 --data-dir $D > faulty.log 2>&1
removed '~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/base/data/default/rules/catalog.json'
removed '~/.no-mistakes/evidence/01M460FVAK2JGE7XSP6HQE5GXW/base/data/default/schema.json'
== BASE after restart ==
> .kg use default
Switched to knowledge graph: default
> .rule list
No rules defined.
> .rel
No relations in current knowledge graph.
> +item[(1), (2)]
Inserted 2 fact(s) into 'item'.
> ?v(X)
No results.

Evidence: Fault-injection shim source (EIO on directory fsync)

// LD_PRELOAD shim: fsync()/fdatasync() on a directory listed in FAIL_FSYNC_DIRS
// (colon-separated absolute paths) returns EIO, as a failing disk would.
#define _GNU_SOURCE
#include <dlfcn.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <unistd.h>
static int should_fail(int fd) {
  struct stat st; if (fstat(fd, &st) || !S_ISDIR(st.st_mode)) return 0;
  const char *dirs = getenv("FAIL_FSYNC_DIRS"); if (!dirs) return 0;
  char link[64], path[4096]; snprintf(link, sizeof link, "/proc/self/fd/%d", fd);
  ssize_t n = readlink(link, path, sizeof path - 1); if (n < 0) return 0; path[n] = 0;
  char buf[8192]; strncpy(buf, dirs, sizeof buf - 1); buf[sizeof buf - 1] = 0;
  for (char *s, *t = strtok_r(buf, ":", &s); t; t = strtok_r(NULL, ":", &s))
    if (!strcmp(t, path)) { fprintf(stderr, "[failfsync] injected EIO on fsync(dir=%s)\n", path); return 1; }
  return 0;
}
int fsync(int fd) { if (should_fail(fd)) { errno = EIO; return -1; }
  static int (*real)(int); if (!real) real = dlsym(RTLD_NEXT, "fsync"); return real(fd); }
int fdatasync(int fd) { if (should_fail(fd)) { errno = EIO; return -1; }
  static int (*real)(int); if (!real) real = dlsym(RTLD_NEXT, "fdatasync"); return real(fd); }

Evidence: Recovery after simulated power loss (fixed vs pre-fix)

FIXED: .rule list -> v (1 clause(s)); ?v(X) -> 1, 2; +item[("not-an-int")] -> rejected (schema recovered)
PRE-FIX: .rule list -> No rules defined.; ?v(X) -> No results.

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

⚠️ **Review** - 1 info

ℹ️ src/storage/metadata.rs:490 - The new test test_save_fails_when_the_directory_sync_fails was inserted between the existing doc comment /// Regression: Metadata save creates parent directories if they don't exist. and the test it describes, test_metadata_save_creates_parent_dirs. The comment now documents the wrong test. Move the new test above that doc comment, or below test_metadata_save_creates_parent_dirs.

✅ **Test** - passed

✅ No issues found.

Live validation: ✅ go - 4 of 5 scenarios driven live against the product

Scenario	Result	Live	Evidence
A failing directory fsync during a catalog save is reported as a failed save, not Ok	✅ pass	live	faulty.log: '[failfsync] injected EIO on fsync(dir=.../default/rules)' then 'WARN catalog_save_failed_wal_keeps_changes ... Failed to write catalog: I/O error: Input/output error (os error 5)', and th…
A catalog-only commit (schema + rule) whose directory fsync failed keeps its changes in the WAL instead of pruning them	✅ pass	live	scenario.log: current.wal is 449 bytes and mentions item after the commit; scenario-base.log (pre-fix): the WAL directory is empty after the same commit
After a power loss drops the un-synced catalog renames, a restarted server still has the committed rule and schema	✅ pass	live	scenario.log: after kill -9 and deleting rules/catalog.json and schema.json, the restart shows 'v (1 clause(s))', ?v(X) returns 1 and 2, the string insert into item is rejected, and the catalog files…
Regression tests fail before the fix and pass after it (fault injection on both the rules dir and the KG dir)	⏸️ untested	no	The prior payload established this only through cargo unit tests using cfg(test) inject_sync_fault, not against the live product, so no live result was recorded for this scenario. The live behaviour i…
Healthy path: when directory fsync succeeds, saved catalog changes still leave the WAL	✅ pass	live	scenario.log: after the fault-free restart rewrote the catalogs, current.wal shrank to 128 bytes; the saved_catalog_changes_leave_the_wal test passes

cargo test --lib -- unsynced_catalog_directory_keeps_the_changes_in_the_wal test_save_fails_when_the_directory_sync_fails saved_catalog_changes_leave_the_wal (all pass)
Same two new tests run with the base-commit save_json_atomic body temporarily restored: both FAIL (regression reproduced), source then restored
Built an LD_PRELOAD shim (failfsync.so) that returns EIO from fsync/fdatasync on chosen directories
Live: inputlayer-server (fixed) under the shim failing fsync on data/default and data/default/rules; inputlayer-client commit.iql registers schema item(x: int) and rule v(X) <- item(X); checked server log and WAL
Live: kill -9 the server, delete default/rules/catalog.json and default/schema.json (renames never made durable), restart without the fault, inputlayer-client check.iql (.rule list, insert, ?v(X)) and schema_check.iql (string into int column)
Live control: the same steps on a server binary built from the base commit's metadata.rs

✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

save_json_atomic ignored the parent-directory fsync error and returned Ok. For a catalog-only commit the WAL then pruned its copy of the changes, so a power loss could undo the rename and silently drop a committed rule or schema. Propagate the error through sync_directory instead, so the WAL keeps the changes until both catalog files are durable. Closes #297

…etadata tests

jsam added 3 commits October 5, 2026 06:47

Merge remote-tracking branch 'origin/main' into fm/fix-catalog-fsync

30142d0

no-mistakes(document): Reattach misplaced regression doc comment in m…

75e1803

…etadata tests

jsam merged commit 6ac5322 into main Oct 5, 2026
20 checks passed

jsam deleted the fm/fix-catalog-fsync branch October 5, 2026 12:47

This was referenced Oct 5, 2026

Engine release tracker: checkpoints 1-2 live status #199

Closed

fix(protocol): keep keyed subscription families sharing when sharing saves most of the work (#378) #385

Merged

Navigation Menu

Repository navigation

fix(storage): fail catalog saves when the directory fsync fails so committed rules stay in the WAL - #333

fix(storage): fail catalog saves when the directory fsync fails so committed rules stay in the WAL#333
jsam merged 3 commits into
mainfrom
fm/fix-catalog-fsync

jsam commented Oct 5, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Conversation

jsam commented Oct 5, 2026

Intent

What Changed

Risk Assessment

Testing

Pipeline

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees