Skip to content

Epic: persistent rules are incrementally maintained views; everything else is a query (E0) #305

Closed

Description

@jsam

Problem. Persistent rules are stored definitions that every read recompiles and re-derives from base facts in a fresh timely dataflow (src/storage_engine/snapshot.rs:302-405, src/lib.rs:1481-1775, src/code_generator/mod.rs:390,707,1079). Subscriptions re-run their query per relevant commit and diff full results (src/protocol/subscription/reevaluate.rs:124-153). The "incremental engine" mirrors base relations only, is never enabled on the server path, and no rule is compiled into it (src/storage_engine/mod.rs:2318-2350, src/storage_engine/catalog_change.rs:424-429). Measured on the benchmark host at 1M edges: one-row bound read of a view 62 ms (non-recursive) / 2.9 s (recursive); write->delta with one subscriber 70 ms / 3.0 s; ten keyed subscribers on a recursive view 30 s CPU per write and 11.5 GB RSS. A differential-dataflow maintainer of the same rules applies a change in ~100 µs at every size. The docs claim the maintained behaviour.

Target. One per-knowledge-graph view maintainer (long-lived differential dataflow fed from the commit path at the commit's revision) holds every persistent rule as an arrangement; queries import arrangements at a stated revision and never re-derive; subscriptions are keyed diffs of a view between revisions; the cost probe disappears; docs tell the truth. Report Sections 1, 2 and 5.

Acceptance. Every benchmark in report Section 9 (B1-B23, including the vector bars B17-B23) passes in the perf gate on the benchmark host; the differential oracle shows 0 divergences across recursion, negation, aggregates and deletes; rule_evaluations is 0 for reads of deployed views; the docs-truth issue is closed; engine.views default is maintained and the recompute path for persistent rules is deleted.

Children (train order). Slot 1: D1, V0, V1. Slot 2: V2, V3, V4. Slot 3: V5, V6, V7. Slot 4: V9, V13, V10. Slot 5: V11, V12, V14. Slot 6: V15, V16, V18. Slot 7: VV1, VV2, VV3. Slot 8: VV4, VV5, VV6. Slot 9: VV7, V19, V20.

Supersedes. #143 (and its children #180-#187 map one-to-one onto V2, V3, V4, V5, V5, V7, V10, V9+V20); the DD remainder of #133; #238 (bar kept as B15); #276, #289 and #252 (probe-machinery defects removed by V11). Keeps #292 open until B15 passes. Depends on #296 (runaway-query fix) merging before V9. Size: epic. Order: 0.

Activity

  1. jsam commented on Oct 5, 2026

    @jsam
    ContributorAuthor

    Children in merge-train order (max 3 in review at once)

  2. jsam commented on Oct 5, 2026

    @jsam
    ContributorAuthor

    Captain requirement (2026-10-05): every change in this milestone must be covered by crafted end-to-end scenarios that exercise the changes together over the real server and wire protocol, not only in isolation (subscribe to a deployed rule for a key, query at the same revision, write with expect_revision, read-your-writes through views, rule replacement while subscribed, retraction through recursion and negation, vectors in live rules, refused arbitrary-query subscriptions, restart/failover, limits, multi-tenant isolation, many agents on one rule). The scenario catalogue and harness are being defined now; each child issue's acceptance will reference the scenarios it must pass, and the same scenarios run against today's recompute path first so they prove the migration preserves behaviour.

  3. jsam commented on Oct 10, 2026

    @jsam
    ContributorAuthor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions