Repository navigation
Epic: long-lived per-KG incremental dataflows (real streaming reasoning) #143
Description
Every query today rebuilds timely dataflows over full relations. Materializations are dropped on the first write, and the shadow IncrementalEngine is never read. Goal: one worker per KG holding base arrangements and persistent-rule dataflows, with writes applied as deltas plus advance and step. Bound queries read arrangements by key, and standing queries attach DD output deltas. Gate on the differential oracle (incremental vs recompute vs naive) and voicestack latency benches (p99 bound lookup, write-to-delta lag).
Delivery plan
R4 follows the oracle and atomic engine contract. Reuse the existing incremental worker and canonical IR; storage remains authoritative and unsupported operations keep a correct recompute fallback.
- Testing: revision-aware differential correctness oracle #167 — R1 prerequisite: differential correctness oracle
- Dataflow: Revision-aware per-KG worker ownership #180 — D01 — Revision-aware per-KG worker ownership
- Dataflow: Persistent positive rule graphs and arrangements #181 — D02 — Persistent positive rule graphs and arrangements
- Dataflow: Persistent recursive SCC maintenance #182 — D03 — Persistent recursive SCC maintenance
- Dataflow: Persistent stratified negation #183 — D04 — Persistent stratified negation
- Dataflow: Persistent aggregates and explicit operator capability coverage #184 — D05 — Persistent aggregates and explicit operator capability coverage
- Dataflow: Atomic rule-generation rebuild and swap #185 — D06 — Atomic rule-generation rebuild and swap
- Dataflow: Attach standing queries to frontier-complete DD output #186 — D07 — Attach standing queries to frontier-complete DD output
- Dataflow: Serve arranged lookups and adopt persistent dataflow by default #187 — D08 — Serve arranged lookups and adopt persistent dataflow by default
- Standing queries: share evaluations, debounce, then DD-backed deltas #133 — shared evaluation in R3, DD-backed standing-query completion in R4
Prerequisites also include #135 resource lifecycle and #163 performance gates. Every supported operator/frontier must pass the oracle, real SDK delivery and crash/soak checks before default adoption. Do not close on a shadow-worker demonstration alone.
Activity
jsam commentedon Oct 4, 2026
Moved to Later / deferred. Epic for persistent incremental dataflow (R4): follows engine production readiness.
The neurosymbolic decision layer plan narrows the roadmap to two milestones: Engine production readiness (current focus), then Gateway neural integration (Jev, CLM, Laya) (#266). This issue stays open and can be pulled back in. Tracker: #199.
jsam commentedon Oct 5, 2026
Superseded by #305 (E0) in the incremental-views plan, epic #305.