feat(ws): ordered notification stream with epoch cursor and revision-tagged subscriptions (#136) - #215

…terval

Each round's two arms run back to back, so judge the median of the
per-round cost ratios; host drift between rounds cancels. The exact
binomial interval of a median replaces the bootstrap: deterministic,
and at 10 rounds it tolerates one outlier round per side. Agents now
drain against a deadline from the writer's last write, so a slow writer
no longer counts against them.

QueryJob::execute holds a storage read guard for the whole program, but
.debug, .why, .why full, .why_not and the .index commands acquired a
second read on the same lock. parking_lot's RwLock prefers writers, so a
writer queued between the two reads deadlocked the query.

- .debug and .index reuse the outer guard.
- .why and .why_not capture a ProofSnapshot (KG snapshot plus index
  metrics) under the outer guard, release it, then run query evaluation
  and proof search on the snapshot alone. A failed proof re-acquires the
  guard for the statements that follow, as .compact does.
- StorageEngine::proof_snapshot_on and KnowledgeGraphSnapshot::proof_inputs
  separate snapshot capture from evaluation; execute_and_get_context is
  built from them and the unused get_rules_and_data is removed.

A test-only hook queues a writer after execute takes its guard and before
the meta command runs; every covered command deadlocked before this fix
and completes after it. No lock is added.

Sessions authenticated over /ws kept working for up to 24 h after their
API key was revoked, their password changed or their user dropped:
execute_program only re-read the role, and queued results, notifications
and subscription pushes were sent without any credential check.

Credentials now live in an in-memory CredentialRegistry, loaded from
_internal at startup and updated by every user and key mutation after
it is persisted. Authentication returns a Principal bound to exactly one
credential: one API key, or one generation of a user's password, each
with a canonical CredentialId. Revoking a credential trips its flag
before the command returns; other credentials, including other keys of
the same user, are untouched.

Enforcement:
- execute_program checks the principal at admission (one immutable
  permission snapshot for the whole program, replacing the per-request
  users scan) and again at release, so a credential revoked mid-run gets
  none of the output; writes committed before revocation stay.
- every outbound /ws data frame passes the fence after the socket is
  ready and before the frame is committed to it;
- a per-connection one-shot revocation signal closes the connection,
  stops its standing queries and sends one final notice.
- kg_change notices for other KGs reach admins only, and _internal
  changes are never sent, live or on replay.

Checks are an atomic load on state the session already holds; no shared
lock, lookup or storage access per request or frame. Login and REST key
checks use one registry lookup instead of a snapshot scan.

Closes #137

Adds `perf-gate genbi` and `make bench-genbi`: each genbi-trust scenario
(read in place from GENBI_TRUST_DIR, never copied) runs on a fresh server.
The IQL seed is cold-loaded, every business question becomes a standing
subscription over /ws, and an independent writer applies the ordered
mutations. Per mutation it records writer-send to agent-delta latency
(split insert vs retraction, and propagation after the durable ack),
write ack, a full re-query baseline, memory and throughput; every
checkpoint compares the agent's maintained answer with the
evaluator-private expected rows (counts only are reported) and with a
fresh evaluation, so a broken delta path fails as delta divergence.
--fault drop-retractions proves the checks catch one.

The perf-gate client gains rows-returning query() and subscription ids
on pushed frames; the server helper reports current RSS.

A capped result was adopted as the full set: .subscribe returned it as a
complete snapshot (truncated: false) and a re-evaluation crossing the cap
diffed against the partial rows. Reject a truncated result before touching
state: .subscribe errors without registering, re-evaluation pushes a
subscription_error and keeps the last complete result, and the next delta
once the result fits is relative to it.

Both SDKs turned an engine `error` frame into a one-row result with an
`error` column, and ignored `errors[]`, so a failed insert, load or
migration read as success (Python insert returned InsertResult(count=1)
for a WAL append failure, #93).

One error contract per SDK, applied in the connection:
- `error` frame -> QueryError with `code` and validation errors.
- non-empty `errors[]` -> StatementFailedError with each failed
  statement and the whole result; chunked results keep `errors` and
  `timing_breakdown` through assembly, and an error frame mid-stream
  raises typed.
- KG auto-create, migration recorder and langchain/langgraph helpers
  branch on the exception and `code`, never on message text.
- insert() counts the facts the engine reports storing.

JS fixes the error contract exposed:
- the reader queues a reply's frames, so a chunked reply sent in one
  burst is no longer dropped after its first frame;
- load() sends the local file instead of the client-only `.load`;
- vectorSearch() sends a plain query and sorts and limits locally
  instead of a head aggregate the engine rejects;
- dropKnowledgeGraph() leaves the KG before dropping it.

Two servers on one data_dir (for example a Kubernetes RollingUpdate on a
ReadWriteOnce volume) interleave WAL appends and shard rewrites and corrupt
both. StorageEngine::new now takes an OS advisory lock (flock / LockFileEx
via fs4) on data_dir/LOCK before creating, reading or writing anything
else, and holds it as the engine's last-dropped field.

A second engine on a locked directory fails with
StorageError::DataDirLocked naming the directory and the owner's recorded
pid, and leaves the directory byte-identical. The OS releases the lock when
the owner exits, including SIGKILL, so a stale LOCK file never blocks a
restart. Distinct directories lock independently.

Startup only: one open + flock + pid write (~80us measured); no per-request
lock or hot-path change.

Closes #142

Add `make e2e-reactive`: each test starts a real inputlayer-server
process, agents subscribe to standing queries over /ws, and independent
writer connections insert and retract facts and change rules. Agents
must receive the exact added/retracted rows with contiguous seq, without
re-querying, and end equal to a fresh full query. Covers one subscriber,
64 subscribers, reconnect/resubscribe and crash-restart, unrelated
writes, and write bursts.

Defects tracked by the reactive plan run as expected failures that fail
as XPASS once fixed: capped snapshots and refreshes adopted as complete
(S05), out-of-order notifications and a restart-reused reconnect cursor
(W04), and an undeliverable oversized delta that still advances the
subscription (W05).

The harness is the test-only `testkit` workspace crate (engine process,
/ws agent client, fixtures, raw writer->agent latency samples in schema
inputlayer.reactive.delta_latency.v1), shared with the benches. The
suite runs in test-all, ci-test-all and lint; plain `cargo test` skips it.
No production code changes.

A write now commits a Transaction: every change of one commit, across any
number of shards, at one revision. The WAL stores it as one checksummed
record whose trailing newline is the commit boundary, written with one
append and (in immediate mode) one fsync. Recovery replays the longest
intact prefix of records, so a restart yields the state after an exact
number of commits, never part of one.

- PersistBackend::append(shard, updates) becomes commit(txn). Inserts and
  deletes commit one-op transactions; startup multiplicity corrections and
  prefix clears commit one transaction across all their relations, so a
  failed prefix clear now leaves every relation intact.
- TxnOp is a typed enum (facts today); rule and schema changes become new
  variants. A checksummed record this server cannot decode fails startup
  with StorageError::WalUnreadable instead of being skipped.
- Tuples are stored with the lossless batch codec (base64), fixing NaN and
  infinite floats that JSON wrote as null and replay then dropped. v1
  records still replay so v1 data dirs migrate with their WAL.
- A crash-torn last line is cut off. Any other damage stops replay there
  and the cut-off bytes are saved to wal/current.wal.<ms>.corrupt, instead
  of skipping the damaged record and replaying later ones out of order.
- A failed write or fsync is cut back to the exact committed length; if
  that fails, the repair runs before the next write, rewrite or shutdown,
  so a commit that returned an error is not replayed.
- Remove the unused legacy src/storage/wal.rs, a second WAL format.

Hot path: same locks (WAL mutex, shard map, per-KG write lock), nothing
on the query path; one record per commit instead of one per tuple.

Part of #131

…tions (#88)

Replay every corpus.json scenario, controls included, and every
benchmark.json recorded reference extraction through the real rule pack,
and score both with one evaluator contract. Prints per-family detection,
the false-alarm rate over control rows and revision fidelity over
correction rows; fails on any control finding, stale correction,
over-retraction, finding-kind drift, unevaluated scenario, vacuous
control, missing family or empty metric. No model calls.

- evaluator.py: pure verdict/metric/contract module, self-tested by
  test_evaluator.py with planted defects
- engine_replay.py: corpus facts -> one namespaced KG, two-phase
  assert/retract, per-scenario observation with cross-scenario leak check;
  replaces the duplicate full_bench.engine_pass
- gate.py + scripts/run_vc_gate.sh + make vc-gate: CI entry on a
  throwaway engine; path-filtered workflow verified-completions.yml
- corpus controls now carry extractor-truth facts (corrections retract
  the superseded fact); fixes six restatement controls whose "$1k" vs
  "1500 USD" was a real contradiction
- poc_verify scores recorded extractions through the shared contract and
  checks presence against the engine; verify_each no longer passes
  controls without evaluating them

Add tests/differential_oracle: one history (statements, restarts, named
checkpoints) replayed through independent adapters and compared at every
checkpoint as Z-sets:

- reference: naive stratified fixpoint over finite sets in the test,
  sharing only the parser with the engine; unmodeled constructs desync it
  with an explicit, reported reason
- recompute: the snapshot evaluator queried afresh
- subscription: standing queries assembled purely from pushed deltas via
  the real notification, dependency-filtering and coalescing registry
- spec: results recorded in .iql.out transcripts (corpus cases)

Histories come from hand-written scenarios (duplicate supports, recursive
edge removal, negation, aggregates, rule replacement, restart), seeded
random generation and 620 .iql.out cases. Divergences are minimized with
delta debugging; a deliberately corrupted adapter (dropped retractions) is
caught and shrunk to two statements. The Adapter trait takes the revision
an answer must reflect, so a persistent per-KG dataflow adapter plugs in.

Production change: the notification-to-ChangeSet mapping moves into
subscription::change_of so the oracle drives the same mapping the
WebSocket loop uses. Add `make oracle-test` and document it in TESTING.md.

S05 landed in this checkpoint, so its two expected-failure markers flip
to required passes that pin the exact contract: a capped snapshot is
refused with nothing registered, and a refresh over the cap pushes
subscription_error, keeps the last complete result and recovers with a
contiguous delta.

Credential revocation (S03) shares subscription re-evaluation with the
fail-closed result cap (S05) and the handler with the guard-free proof
path (S06). One test holds a subscriber at the cap and revokes it: the
recovery delta never reaches it. The other revokes during .why proof
search, which now runs after the storage guard is released: no proof
is delivered.

make doc-check (rustdoc -D warnings) rejects intra-doc links from public
module docs to private modules; name them in plain text instead.

Quiet-window A/A of the approved baseline at 20 rounds with pinned
servers: 17 of 20 required metrics pass against themselves; durable-ack
p50 and interference delta p99 stay inconclusive on this host's virtual
disk. Raw samples and host notes sit beside approved.toml.

The 2,000-row .why full proof took 38-48 ms in release, below the
100 ms needed to revoke during it, so the test failed there. Grow the
fixture until the fastest of two proofs takes 100 ms, and revoke at a
quarter of that, so it holds in debug and release.

A program that changes persistent state now commits all of it as one
transaction on its knowledge graph: facts, schema declarations, persistent
rules and rule removals (-name, .rule drop, .rule drop prefix, .rule
remove, .rule clear). Statements apply in order to staged copies of the
catalogs under the KG's existing write lock, so a fact is checked against
a schema declared earlier and a conditional delete or update after a rule
change evaluates with the new rule. Everything is written as one WAL
record and published as one snapshot: readers see one rule generation
with its data. The first failing statement fails the program; nothing is
applied, its query does not run, and subscribers are not notified.

Commands that cannot join the transaction (.kg, .rel, .rel drop, .clear,
.index, .compact, inspection, user, ACL, ontology, subscription) are
rejected with code unsupported before any statement runs when they share
a program with writes. Programs without writes run as before.

- TxnOp::Catalog carries each changed rule's and persistent schema's
  state after the commit, so replay is idempotent. Catalog files record
  the revision they reflect and are written atomically; the WAL keeps a
  catalog change until both files are saved past it, and startup replays
  what the files miss. KG drop removes its WAL catalog changes.
- Every catalog API is a one-statement program: one write path.
- .ontology install deploys as one transaction; .ontology remove drops a
  pack's rules in one transaction and each relation on its own.
- Python SDK load() docs state the real transaction boundary.

Hot path: fact commits take the same locks as before (the catalog log
lock only when a transaction carries catalog changes); queries are
untouched. Rule and schema changes now cost a WAL fsync and two atomic
catalog saves.

Part of #131 (slice #170).

# Conflicts:
#	.github/workflows/ci.yml
#	Cargo.toml
#	docs/content/docs/guides/persistence.mdx
#	docs/guides/persistence.md
#	packages/inputlayer-py/src/inputlayer/knowledge_graph.py
#	src/protocol/handler.rs
#	src/storage/persist/mod.rs
#	src/storage/persist/transaction.rs
#	src/storage/persist/wal.rs
#	src/storage/persist/wal_record.rs
#	src/storage_engine/mod.rs
#	tests/wal_transaction_recovery.rs

…ed subscriptions

Change notifications and standing-query results now have explicit, separate
stream contracts.

- One NotificationLog per engine run numbers, retains and broadcasts each
  notification under its existing ring lock, so seq order, ring order and
  delivery order agree; the two duplicated publish paths and their three
  fields are gone.
- authenticated carries stream_epoch, a random id of the engine run. A
  reconnect presents last_seq with epoch; the server replays the retained
  notifications after it with live ones following without gap or overlap,
  or sends one non-closing replay_gap notice (other epoch, evicted, ahead).
  Replay passes the credential fence frame by frame.
- Snapshots carry a revision from one engine-wide counter, assigned under
  the KG write lock. A subscription evaluation pins one snapshot (rules,
  data and revision) via Handler::query_snapshot; subscribed and every
  subscription_delta name the revision they are the exact answer at.
- Registration compares the KG's current revision with the snapshot's and
  re-evaluates at once when it moved on, so a commit announced before the
  subscription existed is not lost.

Hot path: no new lock. Publication extends the existing ring critical
section by the broadcast send; subscribe adds one snapshot read.

Part of #174, #136

…ng-subscribe delivery

The W04 known-defect markers become required passes in a new stream module:
concurrent commits notify in strictly increasing seq order; a cursor from
before an engine restart gets one replay_gap notice and no replayed
notifications of the new run; and commits racing a .subscribe all reach the
agent, whose view ends equal to a fresh query.

The testkit agent now checks that every delta names a higher revision than
the state it applies to (Violation::StaleRevision) and exposes the stream
epoch. In-process tests cover the reconnect cursor over a real /ws
connection (replay then live without overlap; other or missing epoch and
evicted cursors get replay_gap) and increasing revisions on deltas.

Part of #174

The AsyncAPI spec, the WebSocket guide, the ws-protocol crate docs and
TESTING.md describe the two stream contracts: notification seq within a
stream epoch with last_seq+epoch replay or replay_gap, and per-subscription
gapless seq with increasing knowledge graph revisions. The Python and JS SDK
and GUI frame types gain stream_epoch, Subscribed.revision,
SubscriptionDelta.revision and the replay_gap notice code (non-closing).
The GUI docs bundle is regenerated, which also picks up earlier guide
changes it had not yet bundled.

Part of #174, #136

A session kept receiving its knowledge graph's change notifications, live and
on replay, after .kg acl revoke took its access away, and a subscription
delta computed before the revocation could still be sent.

Every pushed frame now passes a read-access check for its knowledge graph.
The handler counts access-list changes (grant, revoke, KG drop cleanup, user
drop); each connection caches its answer per knowledge graph and looks the
access list up again only when that generation moved. A withheld delta
becomes a subscription_error naming its skipped seq.

Hot path: one atomic load per pushed frame while access lists are unchanged.

Part of #174

# Conflicts:
#	Cargo.toml
#	src/protocol/handler.rs
#	src/storage/persist/mod.rs
#	src/storage/persist/transaction.rs
#	src/storage/persist/wal.rs
#	src/storage_engine/catalog_commit_tests.rs
#	src/storage_engine/program_commit_tests.rs
#	tests/atomic_program_tests.rs
#	tests/fact_program_tests.rs