Skip to content

perf(neo4j): use CREATE instead of MERGE for CALLS edges in write_function_call_groups - #1161

Merged
Shashankss1205 merged 1 commit into
CodeGraphContext:mainfrom
ModQA:perf/neo4j-calls-create-instead-of-merge
Jun 10, 2026
Merged

Shashankss1205 merged 1 commit into
CodeGraphContext:mainfrom
ModQA:perf/neo4j-calls-create-instead-of-merge

Conversation

@ModQA

@ModQA ModQA commented Jun 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Fixes a performance regression in write_function_call_groups() (writer.py) where CALLS relationship writes use MERGE instead of CREATE on Neo4j. This causes ~130ms latency per edge due to exclusive lock acquisition, making large repos (35k+ CALLS edges) take 60–90 minutes for the post-processing phase and hitting CI timeouts.

  • Switches to CREATE for Neo4j/Nornic backends — safe because all call paths delete existing CALLS edges before this method is called (delete_repository_from_graph for --force reindex; delete_outgoing_calls_from_files for incremental watch updates)
  • Non-Neo4j backends (KuzuDB, FalkorDB) retain MERGE — KuzuDB's UNWIND fallback path re-executes rows individually, making CREATE unsafe there
  • Also fixes 6 pre-existing ruff violations in the two changed files, and corrects two pre-existing test failures in TestDeleteRepositoryFromGraph whose assertions described an unimplemented backslash fallback path

Background

This restores the speedup originally introduced in PR #796 (perf/fix: V3 CALLS batching), which used CREATE and reported ~1400× speedup (6 hours → ~16 seconds on a 28K-file Java repo). That fix was in graph_builder.py. When the code was refactored to extract GraphWriter, write_function_call_groups() was reimplemented from scratch using MERGE, silently losing the optimisation.

Closes #1159

Observed impact (self-hosted Neo4j Community 5.15.0)

Repo CALLS edges Write time (MERGE)
Service 1 (300 files) 2,562 335 seconds
Service 2 (1,780 files) 35,466 hit 30-min CI timeout

With CREATE, Neo4j no longer needs to acquire exclusive locks on both endpoint nodes per edge — write time drops to seconds.

Changes

  • src/codegraphcontext/tools/indexing/persistence/writer.py — detect backend at start of write_function_call_groups, use CREATE for Neo4j/Nornic, MERGE for all others; fix 4 pre-existing ruff violations (Tuple unused import, two unused variables, bare f-string)
  • tests/unit/tools/test_graph_builder_perf_fixes.py — update test_uses_merge_for_calls_rel → test_uses_create_for_calls_rel_on_neo4j; add test_uses_merge_for_calls_rel_on_kuzudb and test_uses_merge_for_calls_rel_on_falkordb; fix test_calls_merge_identity_excludes_mutable_metadata to work with either keyword; replace two never-passing backslash fallback tests with tests matching actual implementation behaviour; fix 2 pre-existing ruff violations (unused imports)

Test results

53 passed in 0.72s

All checks passed (ruff).

@vercel

vercel Bot commented Jun 9, 2026

Copy link
Copy Markdown

@ModQA is attempting to deploy a commit to the shashankss1205's projects Team on Vercel.

A member of the Team first needs to authorize it.

…ction_call_groups

MERGE on Neo4j relationships acquires an ExclusiveLock on both endpoint
nodes per edge to check for existing relationships. With no index on
relationship properties this costs ~130ms/edge, making large repos
(35k+ CALLS edges) take 60-90 minutes for the post-processing phase.

Every call path that reaches write_function_call_groups first deletes
existing CALLS edges for the affected scope:
  - full reindex (--force): delete_repository_from_graph clears the repo
  - incremental watch update: delete_outgoing_calls_from_files clears callers

With no existing edges to guard against, MERGE's lock overhead is pure
waste. Switching to CREATE drops write time from ~130ms/edge to
~1-2ms/edge (restoring the speedup originally introduced in PR CodeGraphContext#796,
which was lost when graph_builder.py was refactored into writer.py).

Non-Neo4j backends (KuzuDB, FalkorDB) retain MERGE: KuzuDB's UNWIND
fallback path re-executes rows individually, making CREATE unsafe there.

Closes CodeGraphContext#1159

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@ModQA
ModQA force-pushed the perf/neo4j-calls-create-instead-of-merge branch from be173a1 to 6c2a9ad Compare June 9, 2026 15:28
@Shashankss1205
Shashankss1205 merged commit ef00899 into CodeGraphContext:main Jun 10, 2026
1 check failed
@github-project-automation github-project-automation Bot moved this from Backlog tasks to Done in CGC Progress Board Jun 10, 2026
ModQA added a commit to ModQA/CodeGraphContext that referenced this pull request Jun 11, 2026
…tion scope, and MATCH optimisation

Three compounding issues caused slow CALLS edge writes on Neo4j.

1. Broken backend detection in write_function_call_groups: the legacy
   getattr chain returned the method object (always truthy) instead of
   calling it, silently using MERGE instead of CREATE on neo4j/nornic
   backends — negating PR CodeGraphContext#1161 entirely. Replaced with the canonical
   get_backend_type() call already used elsewhere in the file.

2. Single managed transaction across all CALLS batches (introduced by
   PR CodeGraphContext#1171): execute_write wrapped the entire _work closure — all 100k+
   edges in one transaction. Neo4j held write locks and accumulated WAL
   entries until the final commit, producing 46-minute silent gaps with
   ~167ms/edge observed in production logs. Replaced with a plain session
   so each 1000-edge batch gets its own execute_write — per-batch
   auto-retry and clean rollback without the lock contention.

3. Inefficient MATCH on called nodes: the called MATCH used only
   name+path (2 of 3 composite index properties), causing Neo4j to scan
   all matching nodes and filter line_number in a WHERE clause — 26,680
   DbHits per lookup vs 2 DbHits with the full composite index. Split
   each batch into fast-path (called_line_number known: full 3-property
   composite index hit) and slow-path (ambiguous resolution: 2-property
   prefix scan, correct behaviour preserved).

   Same fix applied to post_resolution.py where the caller MATCH had no
   label (full node scan, no index possible) and the called MATCH had the
   same 2-property issue.

4. Fixed NameError in graph_builder.py where cgcignore_path was
   referenced in _build_graph_from_scip but never passed as a parameter,
   causing SCIP indexing to always fall back to Tree-sitter.

Benchmarks (~97k fn→fn CALLS edges, Neo4j backend):
  Before (MERGE, single tx):           ~138ms/edge, ~1697s total
  After, 1 repo  (~13k functions):       3.3ms/edge,   320s  (43x faster)
  After, 2 repos (~27k functions, CI):   7.1ms/edge,   689s  (19x faster)

Fast/slow path split: 28% fast (known called_line_number, 2 DbHits),
71% slow (ambiguous resolution, index prefix scan). The fast path
delivers 13,340x fewer DbHits per row — proportionally larger benefit
on graphs with hundreds of thousands of nodes.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
macunha1 pushed a commit to macunha1/CodeGraphContext that referenced this pull request Jun 21, 2026
macunha1 pushed a commit to macunha1/CodeGraphContext that referenced this pull request Jun 21, 2026
…tion scope, and MATCH optimisation

Three compounding issues caused slow CALLS edge writes on Neo4j.

1. Broken backend detection in write_function_call_groups: the legacy
   getattr chain returned the method object (always truthy) instead of
   calling it, silently using MERGE instead of CREATE on neo4j/nornic
   backends — negating PR CodeGraphContext#1161 entirely. Replaced with the canonical
   get_backend_type() call already used elsewhere in the file.

2. Single managed transaction across all CALLS batches (introduced by
   PR CodeGraphContext#1171): execute_write wrapped the entire _work closure — all 100k+
   edges in one transaction. Neo4j held write locks and accumulated WAL
   entries until the final commit, producing 46-minute silent gaps with
   ~167ms/edge observed in production logs. Replaced with a plain session
   so each 1000-edge batch gets its own execute_write — per-batch
   auto-retry and clean rollback without the lock contention.

3. Inefficient MATCH on called nodes: the called MATCH used only
   name+path (2 of 3 composite index properties), causing Neo4j to scan
   all matching nodes and filter line_number in a WHERE clause — 26,680
   DbHits per lookup vs 2 DbHits with the full composite index. Split
   each batch into fast-path (called_line_number known: full 3-property
   composite index hit) and slow-path (ambiguous resolution: 2-property
   prefix scan, correct behaviour preserved).

   Same fix applied to post_resolution.py where the caller MATCH had no
   label (full node scan, no index possible) and the called MATCH had the
   same 2-property issue.

4. Fixed NameError in graph_builder.py where cgcignore_path was
   referenced in _build_graph_from_scip but never passed as a parameter,
   causing SCIP indexing to always fall back to Tree-sitter.

Benchmarks (~97k fn→fn CALLS edges, Neo4j backend):
  Before (MERGE, single tx):           ~138ms/edge, ~1697s total
  After, 1 repo  (~13k functions):       3.3ms/edge,   320s  (43x faster)
  After, 2 repos (~27k functions, CI):   7.1ms/edge,   689s  (19x faster)

Fast/slow path split: 28% fast (known called_line_number, 2 DbHits),
71% slow (ambiguous resolution, index prefix scan). The fast path
delivers 13,340x fewer DbHits per row — proportionally larger benefit
on graphs with hundreds of thousands of nodes.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@Shashankss1205 Shashankss1205 added gssoc:approved GSSoC validation: counts toward scoring level:advanced GSSoC difficulty: 55 pts contributor / 30 mentor mentor:Shashankss1205 GSSoC mentor attribution: credits reviewing mentor quality:clean GSSoC quality: x1.2 contributor / +5 mentor type:performance GSSoC type bonus: performance labels Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gssoc:approved GSSoC validation: counts toward scoring level:advanced GSSoC difficulty: 55 pts contributor / 30 mentor mentor:Shashankss1205 GSSoC mentor attribution: credits reviewing mentor quality:clean GSSoC quality: x1.2 contributor / +5 mentor type:performance GSSoC type bonus: performance

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

perf: CALLS edge writes regressed to MERGE in writer.py — ~130ms/edge on Neo4j (PR #796 fix lost in refactor)

2 participants