Skip to content

LSH similarity joins in deployed rules (VV4) #326

Closed

Description

@jsam

Part of epic #305.

Problem. A threshold similarity rule today either joins every pair (cross product, refused or ruinous at scale) or uses lsh_bucket-keyed joins that are re-evaluated in full on every commit; nothing recognizes them as maintainable candidate joins, and no guard distinguishes the two shapes.

Target. Deployed rules whose body joins two vector-carrying atoms on lsh_bucket/lsh_probes columns (same table and bits) and filters with a distance predicate compile to maintained joins keyed by bucket with exact verification; rules with a distance predicate over an unkeyed pair are allowed while both candidate relations are at most engine.vector_exact_rows (default 10,000) and refused above it with a hint to add a bucket key; deltas: an embedding change retracts the row's pairs and inserts its new bucket mates' pairs. Hyperplanes stay a pure function of the table index so maintained and recomputed results are identical.

Acceptance. Oracle: maintained LSH views equal recompute exactly on random histories with embedding updates; B19 (delta on one embedding change with bucket occupancy <= 1,000: <= 10 ms p50 at 1M vectors); guard tests; docs (vectors.mdx LSH section) describe the maintained semantics and recall trade-off.

Size M. Dependencies VV1, V5. Order slot 8.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions