Skip to content

Engine: fact inserts return success but store nothing when the data directory vanishes (WAL write failure swallowed) #93

Closed

Description

@jsam

What happened

During benchmark runs (#89), the OS cleaned up the temp directory a dev server was using as its storage data_dir. From that moment:

  • server.log filled with [wal] ERROR ensure_writer: path=.../persist/wal/current.wal, parent_exists=false
  • +claim[...] inserts over the WS API kept returning success to the client
  • ?claim(C, E, A, V) immediately after those inserts returned zero rows

So every insert silently became a no-op while reporting success. For a reasoning engine this is the same failure class as #91: no error, wrong results - a client has no way to know its facts never landed.

Repro

  1. Start the server with --config pointing data_dir at some directory
  2. rm -rf that directory (or its persist/wal subtree) while the server runs
  3. Insert facts over WS - observe success responses
  4. Query them back - observe empty results, and WAL errors in the server log

Expected

Either of these would be fine; silently acking is not:

  • Insert returns an error to the client when the write-ahead path is unavailable
  • Server recreates missing storage directories on demand (it creates them at startup already)

Found on the verified-completions-plan branch during inputlayer/inputlayer-symbolic#145 benchmark work; hit us as an engine pass scoring 0/60 with no error anywhere client-side.

Roadmap checkpoint — 2026-10-03

Current scope: the SDK still converts error frames into result rows (packages/inputlayer-py/src/inputlayer/connection.py:265, packages/inputlayer-js/src/connection.ts:240). S02 is isolated in #164 for R1. Keep this umbrella open until the R10 storage-path disappearance probe also verifies acknowledged facts survive restart, including missing directories and unlinked open files. Recreating a directory alone is not data recovery. T01 (#168) provides the transaction failure/replay boundary. Primary milestone is Ops & resources because that is the whole-issue completion checkpoint.

Activity

  1. added
    bugSomething isn't working
    backendBackend/server related
    on Aug 1, 2026
  2. jsam commented on Oct 1, 2026

    @jsam
    ContributorAuthor

    The engine does return an error on the main insert path. The silent success comes from the SDKs, which turn ErrorResponse into a data row. Remaining:

    1. SDKs raise on errors (wp-py-sdk-correctness, wp-js-sdk-correctness).
    2. .clear prefix ignores persist errors (wp-write-effective-delta).
    3. Recreate wal/, shards/ and batches/ on demand, or report unhealthy.
    4. Define the semantics of a WAL append followed by a failed flush, so the ack matches replay (wp-wal-segmented-group-commit).
  3. added this to the Ops & resources milestone on Oct 3, 2026
  4. added
    sdkPython, JavaScript and Rust clients and CLI
    P0Immediate correctness, data-safety or live-credential blocker
    on Oct 3, 2026
  5. jsam commented on Oct 4, 2026

    @jsam
    ContributorAuthor

    Moved to the Engine production readiness milestone, the current focus (tracker #199). EN-1: a fact insert must never report success when the WAL write failed. P0 data safety.

    Roadmap per the neurosymbolic decision layer plan: engine production readiness first, then gateway neural integration with Jev, CLM and Laya (#266).

  6. jsam commented on Oct 4, 2026

    @jsam
    ContributorAuthor

    Fixed by #274 (merged, CI green): a fact write no longer reports success when the write-ahead log cannot store it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Immediate correctness, data-safety or live-credential blockerbackendBackend/server relatedbugSomething isn't workingsdkPython, JavaScript and Rust clients and CLI

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions