Repository navigation
perf(neo4j): use CREATE instead of MERGE for CALLS edges in write_function_call_groups - #1161
Merged
Shashankss1205 merged 1 commit intoJun 10, 2026
Conversation
|
@ModQA is attempting to deploy a commit to the shashankss1205's projects Team on Vercel. A member of the Team first needs to authorize it. |
…ction_call_groups MERGE on Neo4j relationships acquires an ExclusiveLock on both endpoint nodes per edge to check for existing relationships. With no index on relationship properties this costs ~130ms/edge, making large repos (35k+ CALLS edges) take 60-90 minutes for the post-processing phase. Every call path that reaches write_function_call_groups first deletes existing CALLS edges for the affected scope: - full reindex (--force): delete_repository_from_graph clears the repo - incremental watch update: delete_outgoing_calls_from_files clears callers With no existing edges to guard against, MERGE's lock overhead is pure waste. Switching to CREATE drops write time from ~130ms/edge to ~1-2ms/edge (restoring the speedup originally introduced in PR CodeGraphContext#796, which was lost when graph_builder.py was refactored into writer.py). Non-Neo4j backends (KuzuDB, FalkorDB) retain MERGE: KuzuDB's UNWIND fallback path re-executes rows individually, making CREATE unsafe there. Closes CodeGraphContext#1159 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
ModQA
force-pushed
the
perf/neo4j-calls-create-instead-of-merge
branch
from
June 9, 2026 15:28
be173a1 to
6c2a9ad
Compare
5 tasks done
ModQA
added a commit
to ModQA/CodeGraphContext
that referenced
this pull request
Jun 11, 2026
…tion scope, and MATCH optimisation Three compounding issues caused slow CALLS edge writes on Neo4j. 1. Broken backend detection in write_function_call_groups: the legacy getattr chain returned the method object (always truthy) instead of calling it, silently using MERGE instead of CREATE on neo4j/nornic backends — negating PR CodeGraphContext#1161 entirely. Replaced with the canonical get_backend_type() call already used elsewhere in the file. 2. Single managed transaction across all CALLS batches (introduced by PR CodeGraphContext#1171): execute_write wrapped the entire _work closure — all 100k+ edges in one transaction. Neo4j held write locks and accumulated WAL entries until the final commit, producing 46-minute silent gaps with ~167ms/edge observed in production logs. Replaced with a plain session so each 1000-edge batch gets its own execute_write — per-batch auto-retry and clean rollback without the lock contention. 3. Inefficient MATCH on called nodes: the called MATCH used only name+path (2 of 3 composite index properties), causing Neo4j to scan all matching nodes and filter line_number in a WHERE clause — 26,680 DbHits per lookup vs 2 DbHits with the full composite index. Split each batch into fast-path (called_line_number known: full 3-property composite index hit) and slow-path (ambiguous resolution: 2-property prefix scan, correct behaviour preserved). Same fix applied to post_resolution.py where the caller MATCH had no label (full node scan, no index possible) and the called MATCH had the same 2-property issue. 4. Fixed NameError in graph_builder.py where cgcignore_path was referenced in _build_graph_from_scip but never passed as a parameter, causing SCIP indexing to always fall back to Tree-sitter. Benchmarks (~97k fn→fn CALLS edges, Neo4j backend): Before (MERGE, single tx): ~138ms/edge, ~1697s total After, 1 repo (~13k functions): 3.3ms/edge, 320s (43x faster) After, 2 repos (~27k functions, CI): 7.1ms/edge, 689s (19x faster) Fast/slow path split: 28% fast (known called_line_number, 2 DbHits), 71% slow (ambiguous resolution, index prefix scan). The fast path delivers 13,340x fewer DbHits per row — proportionally larger benefit on graphs with hundreds of thousands of nodes. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
macunha1
pushed a commit
to macunha1/CodeGraphContext
that referenced
this pull request
Jun 21, 2026
Co-authored-by: Cursor <[email protected]>
macunha1
pushed a commit
to macunha1/CodeGraphContext
that referenced
this pull request
Jun 21, 2026
…tion scope, and MATCH optimisation Three compounding issues caused slow CALLS edge writes on Neo4j. 1. Broken backend detection in write_function_call_groups: the legacy getattr chain returned the method object (always truthy) instead of calling it, silently using MERGE instead of CREATE on neo4j/nornic backends — negating PR CodeGraphContext#1161 entirely. Replaced with the canonical get_backend_type() call already used elsewhere in the file. 2. Single managed transaction across all CALLS batches (introduced by PR CodeGraphContext#1171): execute_write wrapped the entire _work closure — all 100k+ edges in one transaction. Neo4j held write locks and accumulated WAL entries until the final commit, producing 46-minute silent gaps with ~167ms/edge observed in production logs. Replaced with a plain session so each 1000-edge batch gets its own execute_write — per-batch auto-retry and clean rollback without the lock contention. 3. Inefficient MATCH on called nodes: the called MATCH used only name+path (2 of 3 composite index properties), causing Neo4j to scan all matching nodes and filter line_number in a WHERE clause — 26,680 DbHits per lookup vs 2 DbHits with the full composite index. Split each batch into fast-path (called_line_number known: full 3-property composite index hit) and slow-path (ambiguous resolution: 2-property prefix scan, correct behaviour preserved). Same fix applied to post_resolution.py where the caller MATCH had no label (full node scan, no index possible) and the called MATCH had the same 2-property issue. 4. Fixed NameError in graph_builder.py where cgcignore_path was referenced in _build_graph_from_scip but never passed as a parameter, causing SCIP indexing to always fall back to Tree-sitter. Benchmarks (~97k fn→fn CALLS edges, Neo4j backend): Before (MERGE, single tx): ~138ms/edge, ~1697s total After, 1 repo (~13k functions): 3.3ms/edge, 320s (43x faster) After, 2 repos (~27k functions, CI): 7.1ms/edge, 689s (19x faster) Fast/slow path split: 28% fast (known called_line_number, 2 DbHits), 71% slow (ambiguous resolution, index prefix scan). The fast path delivers 13,340x fewer DbHits per row — proportionally larger benefit on graphs with hundreds of thousands of nodes. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a performance regression in
write_function_call_groups()(writer.py) where CALLS relationship writes useMERGEinstead ofCREATEon Neo4j. This causes ~130ms latency per edge due to exclusive lock acquisition, making large repos (35k+ CALLS edges) take 60–90 minutes for the post-processing phase and hitting CI timeouts.CREATEfor Neo4j/Nornic backends — safe because all call paths delete existing CALLS edges before this method is called (delete_repository_from_graphfor--forcereindex;delete_outgoing_calls_from_filesfor incremental watch updates)MERGE— KuzuDB's UNWIND fallback path re-executes rows individually, makingCREATEunsafe thereTestDeleteRepositoryFromGraphwhose assertions described an unimplemented backslash fallback pathBackground
This restores the speedup originally introduced in PR #796 (
perf/fix: V3 CALLS batching), which usedCREATEand reported ~1400× speedup (6 hours → ~16 seconds on a 28K-file Java repo). That fix was ingraph_builder.py. When the code was refactored to extractGraphWriter,write_function_call_groups()was reimplemented from scratch usingMERGE, silently losing the optimisation.Closes #1159
Observed impact (self-hosted Neo4j Community 5.15.0)
With
CREATE, Neo4j no longer needs to acquire exclusive locks on both endpoint nodes per edge — write time drops to seconds.Changes
src/codegraphcontext/tools/indexing/persistence/writer.py— detect backend at start ofwrite_function_call_groups, useCREATEfor Neo4j/Nornic,MERGEfor all others; fix 4 pre-existing ruff violations (Tupleunused import, two unused variables, bare f-string)tests/unit/tools/test_graph_builder_perf_fixes.py— updatetest_uses_merge_for_calls_rel→test_uses_create_for_calls_rel_on_neo4j; addtest_uses_merge_for_calls_rel_on_kuzudbandtest_uses_merge_for_calls_rel_on_falkordb; fixtest_calls_merge_identity_excludes_mutable_metadatato work with either keyword; replace two never-passing backslash fallback tests with tests matching actual implementation behaviour; fix 2 pre-existing ruff violations (unused imports)Test results
All checks passed (ruff).