Skip to content

Epic: long-lived per-KG incremental dataflows (real streaming reasoning) #143

Closed as not planned

Description

@jsam

Every query today rebuilds timely dataflows over full relations. Materializations are dropped on the first write, and the shadow IncrementalEngine is never read. Goal: one worker per KG holding base arrangements and persistent-rule dataflows, with writes applied as deltas plus advance and step. Bound queries read arrangements by key, and standing queries attach DD output deltas. Gate on the differential oracle (incremental vs recompute vs naive) and voicestack latency benches (p99 bound lookup, write-to-delta lag).

Delivery plan

R4 follows the oracle and atomic engine contract. Reuse the existing incremental worker and canonical IR; storage remains authoritative and unsupported operations keep a correct recompute fallback.

Prerequisites also include #135 resource lifecycle and #163 performance gates. Every supported operator/frontier must pass the oracle, real SDK delivery and crash/soak checks before default adoption. Do not close on a shadow-worker demonstration alone.

Activity

  1. added
    backendBackend/server related
    epicTracking issue grouping related roadmap work
    P1Required next for reliable reactive delivery and operations
    on Oct 3, 2026
  2. jsam commented on Oct 4, 2026

    @jsam
    ContributorAuthor

    Moved to Later / deferred. Epic for persistent incremental dataflow (R4): follows engine production readiness.

    The neurosymbolic decision layer plan narrows the roadmap to two milestones: Engine production readiness (current focus), then Gateway neural integration (Jev, CLM, Laya) (#266). This issue stays open and can be pulled back in. Tracker: #199.

  3. jsam commented on Oct 5, 2026

    @jsam
    ContributorAuthor

    Superseded by #305 (E0) in the incremental-views plan, epic #305.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Required next for reliable reactive delivery and operationsbackendBackend/server relatedepicTracking issue grouping related roadmap work

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions