Skip to content

perf(kuzu): write Parameter, Module and ExternalClass as node-only then relationship-only batches - #1748

Merged
Shashankss1205 merged 1 commit into
CodeGraphContext:mainfrom
PhysShell:perf/split-node-rel-batches-main
Oct 15, 2026
Merged

Shashankss1205 merged 1 commit into
CodeGraphContext:mainfrom
PhysShell:perf/split-node-rel-batches-main

Conversation

@PhysShell

Copy link
Copy Markdown
Contributor

Problem

EmbeddedSessionWrapper.run sends every UNWIND batch that MERGEs a node and a relationship down a per-row loop.
This is a guard against the Kùzu 0.11.3 mis-binding of repeated node keys (#1605).

Three writers use exactly that shape:

  • add_file_to_graph → Function–Parameter, HAS_PARAMETER;
  • add_file_to_graph → File–Module, IMPORTS;
  • write_inheritance_links._run_external → child–ExternalClass, INHERITS.

On a 2062-file C# repository these three turn about 3400 batches into about 48 000 single-row queries. That is about
half of the total graph-write time.

Change

Each of the three writes now runs two queries. Neither of them hits the guard, and the guard itself is unchanged.

  1. Node-only UNWIND with distinct node keys, so no key repeats inside a batch. The old MATCH stays, so a node is
    created only where the combined query would have created it:
    • Parameter keys are first read under the Function MATCH;
    • Module keeps MATCH File;
    • ExternalClass keeps MATCH child.
  2. Relationship-only UNWIND — MATCH both ends, MERGE the relationship, same SET.

Behaviour kept on purpose:

  • Module metadata. Per field, the first non-null value in the existing sort_import_rows_for_metadata order —
    what the sequential coalesce SETs leave.
  • Repeated IMPORTS key. Rows sharing an IMPORTS relationship key keep the last row's values. A batched
    MERGE … SET keeps the first row of a non-adjacent duplicate key; row by row, the last one wins. Rows with different
    keys are never merged.
  • External inheritance. Label-less rows still fan out over every populated child label.
  • Missing field. A missing confidence_label still defaults to 'INFERRED'.
  • Errors. Any error in the split pair reruns that chunk through the old combined query, i.e. the old per-row path
    with its skip / retry / drop semantics. All MERGEs involved are idempotent.

Diff: writer.py +96 / −7. New tests/unit/tools/test_split_node_rel_batches.py. One query-text assertion in
test_graph_builder_perf_fixes.py now looks for the Module coalesce SETs on the node query.

Evidence (Kùzu 0.11.3, Windows)

Equivalence: before vs after, whole-DB dump — every table and property.

The fixture covers:

  • two occurrences of one function;
  • shared and duplicate args;
  • a missing Function;
  • competing, braced, super::* and pub use imports;
  • a non-adjacent duplicate IMPORTS key;
  • one Module from two files;
  • duplicate and fan-out external parents;
  • a missing child.

Results on the fixture:

  • the graph is identical;
  • the per-row path is taken 11 times before and never after;
  • five deliberately wrong variants are all caught: no occurrence_index, last-non-null metadata, no last-row rule, a
    parent created without its child, no label-less fan-out.

Real C# repository (2062 files):

  • first 301 writer calls: identical whole-DB dump;
  • first 2000 external-inheritance rows on a full graph: identical ExternalClass nodes and edges.

Silent-fallback guard. If the split pair ever starts failing and quietly falls back to the per-row path, the new
tests fail. They were checked against that mutation, and they fail only on the "no forced per-row query" assertion,
because the fallback's result is still correct.

Speed. Exclusive query time, on main vs this commit, same data:

shape sample before after speed-up
Parameter / HAS_PARAMETER 301 writer calls 82.5 s 9.3 s ×8.9
Module / IMPORTS 301 writer calls 34.9 s 7.8 s ×4.5
ExternalClass / INHERITS 2000 rows 15.2 s 0.31 s ×48

The 301-call sample took 171.2 s → 64.0 s in total.

A replay of the complete write workload of that repository (2066 writer calls, fresh graph) went from 8808.5 s to
2209.0 s (×3.99)
with an identical semantic projection. That figure was measured on a development branch that also
carries unrelated SCIP-parser changes below this commit. Those changes do not touch these writers, and on main the
same sample graphs are byte-identical to that branch's.

Tests. tests/unit/tools + tests/unit/core fail on exactly the same 4 pre-existing tests before and after.

Not in scope

  • the wrapper guard itself;
  • other node+relationship shapes (js_imports, module_inclusions, …);
  • UID matching, path normalization, buffer / checkpoint settings.

🤖 Generated with Claude Code

…en relationship-only batches

UNWIND batches that MERGE a node and a relationship in one query are forced by the embedded wrapper onto a
per-row loop (Kuzu CodeGraphContext#1605 guard). Parameter/HAS_PARAMETER, Module/IMPORTS and ExternalClass/INHERITS now run a
node-only UNWIND with distinct node keys, then a relationship-only UNWIND, neither of which hits the guard.
The guard itself is unchanged.

Semantics kept: nodes only where the old MATCH succeeded, first non-null Module metadata, the last row's values
for a repeated IMPORTS key (a batched MERGE+SET keeps the first row of a non-adjacent duplicate), the label-less
external fan-out. On any error a chunk is rerun through the old combined query.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@vercel

vercel Bot commented Sep 29, 2026

Copy link
Copy Markdown

@PhysShell is attempting to deploy a commit to the shashankss1205's projects Team on Vercel.

A member of the Team first needs to authorize it.

@Shashankss1205 Shashankss1205 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three real-Kuzu batching tests pass after normalizing the Windows fixture path on Linux (follow-up patch included). Without that patch the new external-parent test fails. Require backend parity and performance evidence before merging the optimization.

Concrete correction: set REPO = Path("C:/fx3").resolve().as_posix() in tests/unit/tools/test_split_node_rel_batches.py. The current relative Windows-style value does not match normalized stored paths on Linux.

@Shashankss1205 Shashankss1205 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reproduced the reported test defect on this unmodified head: 1 failed, 2 passed; ExternalClass endpoints are empty because C:/fx3 is relative on Linux while persisted paths are normalized. #1771 retains your commits and changes only REPO to Path("C:/fx3").resolve().as_posix() before running the optimization's tests. The corrected writer/identity suite is 65 passed, 1 skipped. Full platform and strict parity CI are running there. Original head is not being merged with the failing fixture.

@Shashankss1205
Shashankss1205 merged commit f842991 into CodeGraphContext:main Oct 15, 2026
1 check failed
@github-project-automation github-project-automation Bot moved this from Backlog tasks to Done in CGC Progress Board Oct 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants