Skip to content

Revisions: one engine with a lost data directory refuses every engine's writes in the process #402

Closed

Description

@jsam

Problem. In a process with several open engines, one engine that has lost its data directory refuses every other engine's writes, and the next revision of an unrelated test panics. The integration tests run many engines per process. On main (2ed4a4a), storage_engine_tests::test_save_all_knowledge_graphs failed once in a full cargo test --workspace --all-features run on 4 cores:

thread 'test_save_all_knowledge_graphs' panicked at src/storage_engine/revisions.rs:87:17:
revision 5767227 is above a bound that cannot be raised: I/O error: No such file or directory (os error 2)

Mechanism (src/storage_engine/revisions.rs, from #390). One process-wide Counter has a bound for every open engine. Counter::next raises every engine's bound once the counter passes the lowest one (reserve_through), and panics if any raise fails.

  • Reopening a data directory moves the counter up to that directory's recorded bound (open: last.fetch_max(floor)). That is one block (2^20) above where it stood when that directory was first opened, so a reopen can carry the counter past the bounds of engines opened since.
  • An engine whose directory is gone can't write its bound, so the raise fails and next panics in whichever test asks next.
    • let (storage, _temp) = ... drops the TempDir before the engine (bindings drop in reverse order), so the directory is removed while the engine is still registered.
    • data_dir_loss removes a live engine's directory on purpose.

Production runs one engine per process, where refusing to issue revisions without a durable bound is the intended behaviour. The cross-engine coupling only exists in tests.

Seen.

  • The baseline panic above.
  • With the storage component binaries merged for Testing: Component binaries (T-07) #341, restart_equivalence::concurrent_assert_retract_memory_matches_disk failed once in about 10 loaded runs: an insert_tuples_into on an intact directory returned Io(NotFound). Every write calls revisions::reserve_ahead (src/storage_engine/mod.rs:415, :851, residency.rs:348), which fails while any engine in the process cannot raise its bound. data_dir_loss removes and moves a live engine's directory on purpose, so Testing: Component binaries (T-07) #341's PR keeps data_dir_loss_tests in its own binary.
  • The drop-order window still remains in every multi-engine binary.

Fix options. A raise that fails for an engine whose data directory no longer exists could close that reservation instead of failing every engine. Or each engine could own its counter, keeping a shared floor so revisions still never repeat within the process. Either way, a test with two engines, one of whose directories is removed, then a reopen and a write on the other, should pass.

Acceptance. That regression test, and test_save_all_knowledge_graphs green over 50 repeated runs of the storage binary.

Activity

  1. jsam commented on Oct 6, 2026

    @jsam
    ContributorAuthor

    Probably also the cause of #401 (oracle: a rule commit refused with ENOENT in one adapter's engine).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions