Repository navigation
HNSW maintenance inside the maintainer at the commit's revision (VV2) #324
Description
Part of epic #305.
Problem. Index maintenance runs synchronously under the KG write lock inside apply_delta (src/storage_engine/program_commit.rs:291-305; src/storage_engine/vector_index.rs:259-318); index_deleted scans the whole relation for surviving rows (:300-310); compaction rebuilds in place with parallel_insert (src/hnsw_index.rs:108), which is nondeterministic (#302); index epochs are not revisions.
Target. The maintainer applies base-relation index updates in the step for the commit: epoch = revision, serial inserts in commit order (reproducible per history), deletes/replacements as tombstones, compaction past the threshold as a background generation (build beside, swap at a revision); IndexView captured per revision so at: reads and subscriptions see one state; index_deleted uses the keyed arrangement.
Acceptance. Index contents equal the base relation at every revision through random histories and compactions; write -> index visible <= 10 ms p50 at 1M x 384-dim (B18); hnsw_nearest read <= 3 ms p50 (B17); two runs of the same history give identical search results; #302 linked.
Size M. Dependencies V2. Order slot 7.
Activity
jsam commentedon Oct 7, 2026
Scope note (2026-10-07): measured on the dev box, building an HNSW index in commit order is 16x slower than the parallel build (343 s vs 21 s at 100k x 384; 1M parallel 322 s). To keep restart and failover fast, initial builds (create over existing rows, rebuild, restart, reclaim) stay parallel; reproducible results per history hold for incremental inserts. Reproducibility across restarts and initial builds moves to the follow-up that persists the built graph with the snapshot (fast and deterministic restart).
jsam commentedon Oct 7, 2026
Scope note for the first change under this issue (engine pull request inputlayer-symbolic#52, opened as Refs, not Closes).
Done there: with
engine.views = "maintained"the view maintainer applies base-relation index changes at each commit's revision, serial inserts in commit order, deletes and replacements as tombstones, compaction as a background generation built beside the one in use; each snapshot captures its index view;index_deletedno longer scans the relation; index contents equal the base relation at every revision (differential oracle against recompute mode, also under concurrent writers).Moved to a follow-up, by decision on 2026-10-07: identical search results across runs now hold for histories of incremental writes (an index created on an empty relation) and from an index's first compaction on, but not right after
.index createover existing rows,.index rebuildor a restart. Those first graphs are built in parallel again, because building one insert after the other is about 16 times slower: 343.3 s against 21.1 s for 100,000 vectors of dimension 384 (1,000,000 in parallel: 321.7 s), so a restart of a 1M index would take on the order of an hour. Determinism for restarts and initial builds comes back with the next item, persisting the built graph with the snapshot (fast and deterministic restart).Not measured yet: B18 (write to index visible <= 10 ms p50 at 1M x 384) and B17 (
hnsw_nearest<= 3 ms p50); they are owed to the vector benchmark lane.jsam commentedon Oct 10, 2026
HNSW maintenance inside the maintainer at the commit's revision landed in https://github.com/inputlayer/inputlayer-symbolic/pull/52, https://github.com/inputlayer/inputlayer-symbolic/pull/84, https://github.com/inputlayer/inputlayer-symbolic/pull/95, and B17 is met. B18 was measured at 35.3 ms against a 10 ms bar. Remaining work continues in https://github.com/inputlayer/inputlayer-symbolic/issues/129.