Repository navigation
perf(kuzu): write Parameter, Module and ExternalClass as node-only then relationship-only batches - #1748
Conversation
…en relationship-only batches UNWIND batches that MERGE a node and a relationship in one query are forced by the embedded wrapper onto a per-row loop (Kuzu CodeGraphContext#1605 guard). Parameter/HAS_PARAMETER, Module/IMPORTS and ExternalClass/INHERITS now run a node-only UNWIND with distinct node keys, then a relationship-only UNWIND, neither of which hits the guard. The guard itself is unchanged. Semantics kept: nodes only where the old MATCH succeeded, first non-null Module metadata, the last row's values for a repeated IMPORTS key (a batched MERGE+SET keeps the first row of a non-adjacent duplicate), the label-less external fan-out. On any error a chunk is rerun through the old combined query. Co-Authored-By: Claude Opus 5.5 <[email protected]>
|
@PhysShell is attempting to deploy a commit to the shashankss1205's projects Team on Vercel. A member of the Team first needs to authorize it. |
Shashankss1205
left a comment
There was a problem hiding this comment.
Three real-Kuzu batching tests pass after normalizing the Windows fixture path on Linux (follow-up patch included). Without that patch the new external-parent test fails. Require backend parity and performance evidence before merging the optimization.
Concrete correction: set REPO = Path("C:/fx3").resolve().as_posix() in tests/unit/tools/test_split_node_rel_batches.py. The current relative Windows-style value does not match normalized stored paths on Linux.
Shashankss1205
left a comment
There was a problem hiding this comment.
Reproduced the reported test defect on this unmodified head: 1 failed, 2 passed; ExternalClass endpoints are empty because C:/fx3 is relative on Linux while persisted paths are normalized. #1771 retains your commits and changes only REPO to Path("C:/fx3").resolve().as_posix() before running the optimization's tests. The corrected writer/identity suite is 65 passed, 1 skipped. Full platform and strict parity CI are running there. Original head is not being merged with the failing fixture.
Problem
EmbeddedSessionWrapper.runsends everyUNWINDbatch that MERGEs a node and a relationship down a per-row loop.This is a guard against the Kùzu 0.11.3 mis-binding of repeated node keys (#1605).
Three writers use exactly that shape:
add_file_to_graph→ Function–Parameter,HAS_PARAMETER;add_file_to_graph→ File–Module,IMPORTS;write_inheritance_links._run_external→ child–ExternalClass,INHERITS.On a 2062-file C# repository these three turn about 3400 batches into about 48 000 single-row queries. That is about
half of the total graph-write time.
Change
Each of the three writes now runs two queries. Neither of them hits the guard, and the guard itself is unchanged.
UNWINDwith distinct node keys, so no key repeats inside a batch. The old MATCH stays, so a node iscreated only where the combined query would have created it:
UNWIND— MATCH both ends, MERGE the relationship, same SET.Behaviour kept on purpose:
sort_import_rows_for_metadataorder —what the sequential
coalesceSETs leave.IMPORTSkey. Rows sharing anIMPORTSrelationship key keep the last row's values. A batchedMERGE … SETkeeps the first row of a non-adjacent duplicate key; row by row, the last one wins. Rows with differentkeys are never merged.
confidence_labelstill defaults to'INFERRED'.with its skip / retry / drop semantics. All MERGEs involved are idempotent.
Diff:
writer.py+96 / −7. Newtests/unit/tools/test_split_node_rel_batches.py. One query-text assertion intest_graph_builder_perf_fixes.pynow looks for the ModulecoalesceSETs on the node query.Evidence (Kùzu 0.11.3, Windows)
Equivalence: before vs after, whole-DB dump — every table and property.
The fixture covers:
super::*andpub useimports;IMPORTSkey;Results on the fixture:
occurrence_index, last-non-null metadata, no last-row rule, aparent created without its child, no label-less fan-out.
Real C# repository (2062 files):
Silent-fallback guard. If the split pair ever starts failing and quietly falls back to the per-row path, the new
tests fail. They were checked against that mutation, and they fail only on the "no forced per-row query" assertion,
because the fallback's result is still correct.
Speed. Exclusive query time, on
mainvs this commit, same data:HAS_PARAMETERIMPORTSINHERITSThe 301-call sample took 171.2 s → 64.0 s in total.
A replay of the complete write workload of that repository (2066 writer calls, fresh graph) went from 8808.5 s to
2209.0 s (×3.99) with an identical semantic projection. That figure was measured on a development branch that also
carries unrelated SCIP-parser changes below this commit. Those changes do not touch these writers, and on
mainthesame sample graphs are byte-identical to that branch's.
Tests.
tests/unit/tools+tests/unit/corefail on exactly the same 4 pre-existing tests before and after.Not in scope
js_imports,module_inclusions, …);🤖 Generated with Claude Code