Skip to content

perf/fix: V3 CALLS batching, watcher fixes, incremental rebuild, delete_repository robustness - #796

Merged
Shashankss1205 merged 13 commits into
CodeGraphContext:mainfrom
k282:feature/perf-watcher-improvements
Apr 5, 2026
Merged

Shashankss1205 merged 13 commits into
CodeGraphContext:mainfrom
k282:feature/perf-watcher-improvements

Conversation

@k282

@k282 k282 commented Apr 3, 2026

Copy link
Copy Markdown
Collaborator

Summary

Performance and correctness fixes for large-repo indexing, watcher reliability, and delete_repository robustness. All changes battle-tested on a 28K-file Java codebase.


Changes

Performance

V3 CALLS creation (graph_builder.py)

  • Replaced per-call MERGE with CREATE + label-specific UNWIND batch writes (1000/batch)
  • New _resolve_function_call helper + V3 _create_all_function_calls with UNWIND batching
  • CALLS creation for 28K-file repo: 6+ hours -> ~16 seconds (~1400x speedup)

Batched add_file_to_graph (graph_builder.py)

  • All node types written via UNWIND — O(1) round-trips per node type per file
  • sleep(0)/50 optimization applied to SCIP index path too

Incremental CALLS/INHERITS rebuild on watch events (graph_builder.py, watcher.py)

  • On file change: rebuild relationships only for the changed file (O(k)) instead of full repo scan (O(n))

Bug Fixes

Memory exhaustion on large repos (watcher.py)

  • all_file_data.clear() after linking pass prevents unbounded memory growth in watch mode

Duplicate CALLS accumulation on watch events (graph_builder.py, watcher.py)

  • Delete existing CALLS/INHERITS before re-linking on watch events to prevent duplicate edges over time

Variable-length CONTAINS path in index check (cli_helpers.py)

  • [:CONTAINS] -> [:CONTAINS*] in index_helper to correctly traverse nested directory hierarchy

delete_repository_from_graph correctness (graph_builder.py)

  • Purge orphaned nodes by path prefix after repo delete
  • Delete relationships first, then nodes to avoid OOM
  • Batched delete (LIMIT 1000 loop) to avoid OPTIONAL MATCH cartesian explosion

watch_helper fallback indexed check (cli_helpers.py)

  • Fall back to indexed-repo check when watch target is not the repo root

Testing

Validated on 28,663-file Java codebase (256K functions, heavy Spring/Apache/JUnit dependencies):

  • CALLS graph build: 6h+ -> ~16s
  • Watch mode: memory stable across 1000+ file-change events
  • delete_repository: completes cleanly on large repos

k282 and others added 11 commits April 3, 2026 09:49
…urce indexing

- Changed MERGE to CREATE for CALLS relationships (eliminates false-positive deduplication)
- UNWIND batch processing for relationship creation (~1400x speedup)
- Label-specific CALLS queries (fn->fn, fn->cls, cls->fn, cls->cls, file->fn, file->cls)
- Inheritance batch processing
- Timing instrumentation for post-processing phases
- Verified on Nexus (124,975 CALLS) and Nucleus (449,582 CALLS)
…ectory hierarchy

The index_helper 'already indexed' check used a single-hop query
(Repository)-[:CONTAINS]->(File) which misses files behind directory
nodes in the hierarchical graph structure. Changed to [:CONTAINS*]
to traverse Repository->Directory->...->File chains correctly.
…ad of O(n)

Previously every file change event triggered:
 - full re-scan of all repo files for imports_map (~28k files for nucleus)
 - full re-parse of all files into all_file_data
 - wipe and recreate ALL CALLS/INHERITS for the entire repo

New approach (O(k) where k = changed file + its callers):
 - query Neo4j BEFORE deleting nodes to find which files call into the changed file
 - update imports_map for the changed file only (O(1) file scan)
 - update_file_in_graph: DETACH DELETE automatically cleans up all incoming/outgoing
   CALLS and INHERITS on the changed file's nodes
 - delete outgoing CALLS only from affected caller files (not the whole repo)
 - re-parse only the affected subset (changed file + callers + inheritors)
 - build file_class_lookup from Neo4j query — no full re-parse needed
 - re-create CALLS/INHERITS for the subset only

New graph_builder.py methods:
  get_caller_file_paths(file_path_str)
  get_inheritance_neighbor_paths(file_path_str)
  delete_outgoing_calls_from_files(file_paths)
  delete_inherits_for_files(file_paths)
  get_repo_class_lookup(repo_path)

_create_all_function_calls gains optional file_class_lookup parameter.
watcher.py gains _update_imports_map_for_file(changed_path).

Tested against local neo4j-cgc — all assertions pass.
…er file

Previously add_file_to_graph made N individual session.run() calls per file:
 - 1 per function node
 - 1 per parameter (×avg params per function)
 - 1 per class node
 - 1 per variable node
 - 1 per import statement
 - 1 per class→function CONTAINS link
 - 1 per nested function CONTAINS link
 - 1 per directory level in the hierarchy
For a Java file with 10 methods × 3 params + 5 imports = ~45 round-trips per file.
For nucleus (~28,600 files) this was over 1 million individual Neo4j transactions.

New approach: one UNWIND batch per node type per file.
 - Functions, Classes, Variables, Interfaces, Macros, Structs, Enums, etc.:
   one UNWIND $batch query per label (replaces N individual MERGEs)
 - Parameters: one UNWIND batch for all function params in the file
 - Class→Function CONTAINS: one UNWIND batch
 - Nested Function CONTAINS: one UNWIND batch
 - Imports (JS): one UNWIND batch
 - Imports (other): one UNWIND batch
 - Ruby modules: one UNWIND batch
 - Ruby module_inclusions: one UNWIND batch

Also removed:
 - Stray info_logger("Executing add_file_to_graph with my change!") on every file
 - info_logger("Processing import: ...") on every import statement
 - Pre-seeded repo_path_str parameter: callers can supply the resolved repo path
   string to skip a DB round-trip (MATCH Repository) per file — used by the
   build_graph_from_path_async loop which knows the repo path upfront.
 - asyncio.sleep(0.01) per file → sleep(0) every 50 files.
   Old: 28,600 × 10ms = ~286s (~5min) of pure dead wait for nucleus.
   New: ~0s overhead (yields to event loop only when needed).

Smoke-tested against local neo4j-cgc (admin123) — all node types and
relationship types verified correct after the rewrite.
CONTAINS* chain delete only removes nodes reachable from the Repository
node. Nodes orphaned by a previous crash or partial index survive and
retain their old CALLS/INHERITS relationships. When the next --force
reindex runs MERGE to recreate nodes, it matches these orphaned nodes
and stacks new CALLS on top of the old ones, producing duplicates
(755k → 1.05M on nucleus).

Phase 2 batches a path-prefix MATCH+DETACH DELETE (5000 nodes at a time)
to catch any orphaned nodes not in the CONTAINS chain, ensuring the graph
is truly empty before the new index is written.
DETACH DELETE on nodes with many relationships causes Neo4j to exceed
transaction memory limits (11.2 GiB on remote:
Neo.TransientError.General.MemoryPoolOutOfMemoryError).

Fix: in Phase 2, delete CALLS/INHERITS/CONTAINS relationships in batches
of 10000 first (both outbound and inbound), then delete the now-bare
nodes in batches of 5000 using plain DELETE (no cascade needed).
@vercel

vercel Bot commented Apr 3, 2026

Copy link
Copy Markdown

@kbhaskarla is attempting to deploy a commit to the shashankss1205's projects Team on Vercel.

A member of the Team first needs to authorize it.

@Shashankss1205
Shashankss1205 merged commit 34eebed into CodeGraphContext:main Apr 5, 2026
6 of 9 checks passed
ModQA added a commit to ModQA/CodeGraphContext that referenced this pull request Jun 9, 2026
…ction_call_groups

MERGE on Neo4j relationships acquires an ExclusiveLock on both endpoint
nodes per edge to check for existing relationships. With no index on
relationship properties this costs ~130ms/edge, making large repos
(35k+ CALLS edges) take 60-90 minutes for the post-processing phase.

Every call path that reaches write_function_call_groups first deletes
existing CALLS edges for the affected scope:
  - full reindex (--force): delete_repository_from_graph clears the repo
  - incremental watch update: delete_outgoing_calls_from_files clears callers

With no existing edges to guard against, MERGE's lock overhead is pure
waste. Switching to CREATE drops write time from ~130ms/edge to
~1-2ms/edge (restoring the speedup originally introduced in PR CodeGraphContext#796,
which was lost when graph_builder.py was refactored into writer.py).

Non-Neo4j backends (KuzuDB, FalkorDB) retain MERGE: KuzuDB's UNWIND
fallback path re-executes rows individually, making CREATE unsafe there.

Closes CodeGraphContext#1159

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
macunha1 pushed a commit to macunha1/CodeGraphContext that referenced this pull request Jun 21, 2026
…r-improvements

perf/fix: V3 CALLS batching, watcher fixes, incremental rebuild, delete_repository robustness
macunha1 pushed a commit to macunha1/CodeGraphContext that referenced this pull request Jun 21, 2026
…ction_call_groups

MERGE on Neo4j relationships acquires an ExclusiveLock on both endpoint
nodes per edge to check for existing relationships. With no index on
relationship properties this costs ~130ms/edge, making large repos
(35k+ CALLS edges) take 60-90 minutes for the post-processing phase.

Every call path that reaches write_function_call_groups first deletes
existing CALLS edges for the affected scope:
  - full reindex (--force): delete_repository_from_graph clears the repo
  - incremental watch update: delete_outgoing_calls_from_files clears callers

With no existing edges to guard against, MERGE's lock overhead is pure
waste. Switching to CREATE drops write time from ~130ms/edge to
~1-2ms/edge (restoring the speedup originally introduced in PR CodeGraphContext#796,
which was lost when graph_builder.py was refactored into writer.py).

Non-Neo4j backends (KuzuDB, FalkorDB) retain MERGE: KuzuDB's UNWIND
fallback path re-executes rows individually, making CREATE unsafe there.

Closes CodeGraphContext#1159

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@Shashankss1205 Shashankss1205 added gssoc:approved GSSoC validation: counts toward scoring level:critical GSSoC difficulty: 80 pts contributor / 50 mentor mentor:Shashankss1205 GSSoC mentor attribution: credits reviewing mentor quality:exceptional GSSoC quality: x1.5 contributor / +10 mentor type:performance GSSoC type bonus: performance labels Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gssoc:approved GSSoC validation: counts toward scoring level:critical GSSoC difficulty: 80 pts contributor / 50 mentor mentor:Shashankss1205 GSSoC mentor attribution: credits reviewing mentor quality:exceptional GSSoC quality: x1.5 contributor / +10 mentor type:performance GSSoC type bonus: performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants