Skip to content

Keyed top-k and within-radius operator for hnsw_nearest in deployed rules (VV5) #327

Closed

Description

@jsam

Part of epic #305.

Problem. hnsw_nearest is resolved outside the dataflow before each rule runs (src/execution/hnsw_resolve.rs:1-65; src/lib.rs:1633-1650), HnswScan inside the dataflow yields nothing (src/code_generator/mod.rs:1375-1387), variable queries are refused in recursive rules, and a view reading an index re-evaluates on every commit (src/ast/dependencies.rs:35).

Target. A keyed operator in the maintainer for hnsw_nearest("idx", QV, K, Id, Dist) and within_radius: per distinct query key the current rows (id, distance) in an arrangement; on query-key insert/change one index search; on candidate insert an exact check against every key's current k-th distance (O(keys) distance computations) and insertion with displacement; on candidate delete/change a refill search for the keys holding it; above engine.vector_topk_keys (default 10,000) an approximate reverse mode searches an index of the query vectors with radius = the largest current k-th distance; exact mode (brute force) below engine.vector_exact_rows; row-level deltas at the commit's revision; semantics documented in vectors.mdx with the event table from report Section 5.10.

Acceptance. Oracle on random histories (query-key inserts/changes, candidate inserts/deletes/updates): LSH/exact-mode results equal recompute; HNSW results meet recall@10 >= 0.95 vs brute force and never contain a row the index could not return; the operator's state equals a fresh evaluation after every event; B20 (candidate insert with 10,000 keys <= 10 ms p50, refill <= 2 ms per affected key); query-vector change <= 3 ms p50.

Size L (split allowed: top-k / within-radius / approximate reverse mode). Dependencies VV2, VV3, V9. Order slot 8.

Activity

  1. jsam commented on Oct 8, 2026

    @jsam
    ContributorAuthor

    Split as the issue allows: the top-k operator lands first (PR to follow). Within-radius is #413, the approximate reverse mode above engine.vector_topk_keys is #414.

  2. jsam commented on Oct 10, 2026

    @jsam
    ContributorAuthor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions