Skip to content

ViewMaintainer: one long-lived differential dataflow per knowledge graph, fed from the commit path (V2) #309

Closed

Description

@jsam

Part of epic #305.

Problem. Writes never reach a maintained computation: shadow_write is a no-op because incremental is None in both constructors (src/storage_engine/mod.rs:1994,2292; program_commit.rs:380,396-413). The existing worker uses its own u64 time unrelated to revisions and a bounded 1024-command channel that blocks writers when it lags (src/incremental.rs:22-23,110).

Target. A ViewMaintainer per KG (started at KG load, stopped at drop): a timely worker thread owning an InputSession per base relation, timestamped with the KG revision; base arrangements by tuple and by key; feed(revision, delta) called from apply_delta under the KG write lock (commit order), then advance + step until the probe reaches the revision; a frontier readable from the handler; trace byte/row metering (map_batches); failure state (a dead maintainer marks the KG views_unavailable; storage stays authoritative, the recompute path serves reads). Behind engine.views = maintained (default recompute); no rules yet.

Acceptance. Oracle: after random insert/delete/duplicate histories the base arrangements equal the relation store at every revision; a maintainer panic is contained (writes still commit, KG flagged, unrelated KGs unaffected); frontier lag metric present; no bounded channel can block the commit path (backpressure is "frontier lags", measured); unit tests for start/stop with KG create/drop/load.

Size M. Dependencies V0. Order slot 2. Supersedes #180.

Activity

  1. jsam commented on Oct 6, 2026

    @jsam
    ContributorAuthor

    Done: the per-knowledge-graph view maintainer (V2) is merged: a long-lived differential dataflow fed from the commit path at each commit's own revision, behind engine.views = maintained with recompute still the default; the oracle, panic containment, frontier lag metric and start/stop tests from the acceptance are in. Rules in the maintainer follow in V3 and V4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions