Skip to content

perf(neo4j): fix CALLS write performance — backend detection, transaction scope, and MATCH optimisation - #1192

Merged
Shashankss1205 merged 1 commit into
CodeGraphContext:mainfrom
ModQA:perf/neo4j-calls-write-optimisation
Jun 11, 2026
Merged

Shashankss1205 merged 1 commit into
CodeGraphContext:mainfrom
ModQA:perf/neo4j-calls-write-optimisation

Conversation

@ModQA

@ModQA ModQA commented Jun 11, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Three compounding issues caused slow CALLS edge writes on Neo4j, producing ~167ms/edge and multi-hour indexing times on large graphs.

Root causes fixed

1. Broken backend detection (writer.py)
The legacy getattr chain in write_function_call_groups returned the method object (always truthy) instead of calling it, silently using MERGE instead of CREATE on neo4j/nornic backends — negating PR #1161 entirely. Replaced with the canonical get_backend_type() call.

2. Single managed transaction across all CALLS batches (writer.py)
PR #1171's execute_write wrapped the entire _work closure — all 100k+ edges in one transaction. Neo4j held write locks and accumulated WAL entries until the final commit, producing 46-minute silent gaps with zero progress logs. Replaced with a plain session so each 1000-edge batch gets its own execute_write — per-batch auto-retry and clean rollback without the lock contention.

3. Inefficient MATCH on called nodes (writer.py, post_resolution.py)
The called MATCH used only name+path (2 of 3 composite index properties), causing Neo4j to scan all matching nodes and filter line_number in a WHERE clause: 26,680 DbHits per lookup vs 2 DbHits with the full composite index. Split each batch into:

  • Fast path (called_line_number > 0): full 3-property composite index hit — 2 DbHits
  • Slow path (called_line_number <= 0): ambiguous resolution, 2-property prefix scan (correct behaviour preserved)

Same fix applied to post_resolution.py where the caller MATCH had no label (full node scan across all nodes) and the called MATCH had the same 2-property issue.

4. SCIP cgcignore_path NameError (graph_builder.py)
cgcignore_path was referenced in _build_graph_from_scip but never passed as a parameter, causing SCIP indexing to always fall back to Tree-sitter with a NameError.

Benchmarks

Measured on a large Java Spring Boot service (~97k fn→fn CALLS edges), Neo4j backend:

Condition Graph size ms/edge fn→fn total
v0.4.18 baseline (broken) 1 repo ~138ms ~1697s
This PR, 1 repo ~13k functions 3.3ms 320s — 43× faster
This PR, 2 repos (CI simulation) ~27k functions 7.1ms 689s — 19× faster

Production baseline (v0.4.18): ~167ms/edge, 46-minute CALLS write gaps with zero progress logging. Fast/slow path split: 28% fast (known called_line_number, 2 DbHits), 71% slow (ambiguous resolution, index prefix scan). The fast path delivers 13,340× fewer DbHits per row — proportionally larger benefit on graphs with hundreds of thousands of nodes.

Test plan

  • All existing unit tests pass (tests/unit/tools/test_graph_writer_transactions.py, tests/unit/core/)
  • Pre-existing failure in test_database_kuzu_kotlin_metadata.py confirmed pre-existing on main, not introduced by this PR
  • Full index run completed successfully with per-batch progress logging
  • CI simulation (service re-indexed against a second large repo populated graph) completed
  • SCIP fallback bug verified fixed — no longer errors with NameError

@vercel

vercel Bot commented Jun 11, 2026

Copy link
Copy Markdown

@ModQA is attempting to deploy a commit to the shashankss1205's projects Team on Vercel.

A member of the Team first needs to authorize it.

…tion scope, and MATCH optimisation

Three compounding issues caused slow CALLS edge writes on Neo4j.

1. Broken backend detection in write_function_call_groups: the legacy
   getattr chain returned the method object (always truthy) instead of
   calling it, silently using MERGE instead of CREATE on neo4j/nornic
   backends — negating PR CodeGraphContext#1161 entirely. Replaced with the canonical
   get_backend_type() call already used elsewhere in the file.

2. Single managed transaction across all CALLS batches (introduced by
   PR CodeGraphContext#1171): execute_write wrapped the entire _work closure — all 100k+
   edges in one transaction. Neo4j held write locks and accumulated WAL
   entries until the final commit, producing 46-minute silent gaps with
   ~167ms/edge observed in production logs. Replaced with a plain session
   so each 1000-edge batch gets its own execute_write — per-batch
   auto-retry and clean rollback without the lock contention.

3. Inefficient MATCH on called nodes: the called MATCH used only
   name+path (2 of 3 composite index properties), causing Neo4j to scan
   all matching nodes and filter line_number in a WHERE clause — 26,680
   DbHits per lookup vs 2 DbHits with the full composite index. Split
   each batch into fast-path (called_line_number known: full 3-property
   composite index hit) and slow-path (ambiguous resolution: 2-property
   prefix scan, correct behaviour preserved).

   Same fix applied to post_resolution.py where the caller MATCH had no
   label (full node scan, no index possible) and the called MATCH had the
   same 2-property issue.

4. Fixed NameError in graph_builder.py where cgcignore_path was
   referenced in _build_graph_from_scip but never passed as a parameter,
   causing SCIP indexing to always fall back to Tree-sitter.

Benchmarks (~97k fn→fn CALLS edges, Neo4j backend):
  Before (MERGE, single tx):           ~138ms/edge, ~1697s total
  After, 1 repo  (~13k functions):       3.3ms/edge,   320s  (43x faster)
  After, 2 repos (~27k functions, CI):   7.1ms/edge,   689s  (19x faster)

Fast/slow path split: 28% fast (known called_line_number, 2 DbHits),
71% slow (ambiguous resolution, index prefix scan). The fast path
delivers 13,340x fewer DbHits per row — proportionally larger benefit
on graphs with hundreds of thousands of nodes.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@ModQA
ModQA force-pushed the perf/neo4j-calls-write-optimisation branch from fce9187 to d16577e Compare June 11, 2026 14:15
@Shashankss1205
Shashankss1205 merged commit 1fef6ba into CodeGraphContext:main Jun 11, 2026
1 check failed
@github-project-automation github-project-automation Bot moved this from Backlog tasks to Done in CGC Progress Board Jun 11, 2026
pull Bot pushed a commit to Mu-L/CodeGraphContext that referenced this pull request Jun 12, 2026
PR CodeGraphContext#1183 accidentally removed incremental watcher relinking, and PR CodeGraphContext#1192's
Neo4j fast/slow CALLS MATCH split caused a 1-edge Kuzu/Ladybug parity gap.
Restore incremental _handle_modification, scope the perf split to Neo4j only,
and align unit-test mocks with execute_write_operation.

Co-authored-by: Cursor <[email protected]>
macunha1 pushed a commit to macunha1/CodeGraphContext that referenced this pull request Jun 21, 2026
…write-optimisation

Fix Neo4j CALLS write performance via backend detection, per-batch
transactions, and composite-index MATCH optimisation. Preserve Parameter
and File special-case queries from main during conflict resolution.

Co-authored-by: Cursor <[email protected]>
macunha1 pushed a commit to macunha1/CodeGraphContext that referenced this pull request Jun 21, 2026
PR CodeGraphContext#1183 accidentally removed incremental watcher relinking, and PR CodeGraphContext#1192's
Neo4j fast/slow CALLS MATCH split caused a 1-edge Kuzu/Ladybug parity gap.
Restore incremental _handle_modification, scope the perf split to Neo4j only,
and align unit-test mocks with execute_write_operation.

Co-authored-by: Cursor <[email protected]>
@Shashankss1205 Shashankss1205 added gssoc:approved GSSoC validation: counts toward scoring level:advanced GSSoC difficulty: 55 pts contributor / 30 mentor mentor:Shashankss1205 GSSoC mentor attribution: credits reviewing mentor quality:exceptional GSSoC quality: x1.5 contributor / +10 mentor type:performance GSSoC type bonus: performance labels Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gssoc:approved GSSoC validation: counts toward scoring level:advanced GSSoC difficulty: 55 pts contributor / 30 mentor mentor:Shashankss1205 GSSoC mentor attribution: credits reviewing mentor quality:exceptional GSSoC quality: x1.5 contributor / +10 mentor type:performance GSSoC type bonus: performance

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants