Repository navigation
perf/fix: V3 CALLS batching, watcher fixes, incremental rebuild, delete_repository robustness - #796
Merged
Shashankss1205 merged 13 commits intoApr 5, 2026
Conversation
…urce indexing - Changed MERGE to CREATE for CALLS relationships (eliminates false-positive deduplication) - UNWIND batch processing for relationship creation (~1400x speedup) - Label-specific CALLS queries (fn->fn, fn->cls, cls->fn, cls->cls, file->fn, file->cls) - Inheritance batch processing - Timing instrumentation for post-processing phases - Verified on Nexus (124,975 CALLS) and Nucleus (449,582 CALLS)
…ectory hierarchy The index_helper 'already indexed' check used a single-hop query (Repository)-[:CONTAINS]->(File) which misses files behind directory nodes in the hierarchical graph structure. Changed to [:CONTAINS*] to traverse Repository->Directory->...->File chains correctly.
…ion on large repos
…umulation on watch events
…ad of O(n) Previously every file change event triggered: - full re-scan of all repo files for imports_map (~28k files for nucleus) - full re-parse of all files into all_file_data - wipe and recreate ALL CALLS/INHERITS for the entire repo New approach (O(k) where k = changed file + its callers): - query Neo4j BEFORE deleting nodes to find which files call into the changed file - update imports_map for the changed file only (O(1) file scan) - update_file_in_graph: DETACH DELETE automatically cleans up all incoming/outgoing CALLS and INHERITS on the changed file's nodes - delete outgoing CALLS only from affected caller files (not the whole repo) - re-parse only the affected subset (changed file + callers + inheritors) - build file_class_lookup from Neo4j query — no full re-parse needed - re-create CALLS/INHERITS for the subset only New graph_builder.py methods: get_caller_file_paths(file_path_str) get_inheritance_neighbor_paths(file_path_str) delete_outgoing_calls_from_files(file_paths) delete_inherits_for_files(file_paths) get_repo_class_lookup(repo_path) _create_all_function_calls gains optional file_class_lookup parameter. watcher.py gains _update_imports_map_for_file(changed_path). Tested against local neo4j-cgc — all assertions pass.
…er file
Previously add_file_to_graph made N individual session.run() calls per file:
- 1 per function node
- 1 per parameter (×avg params per function)
- 1 per class node
- 1 per variable node
- 1 per import statement
- 1 per class→function CONTAINS link
- 1 per nested function CONTAINS link
- 1 per directory level in the hierarchy
For a Java file with 10 methods × 3 params + 5 imports = ~45 round-trips per file.
For nucleus (~28,600 files) this was over 1 million individual Neo4j transactions.
New approach: one UNWIND batch per node type per file.
- Functions, Classes, Variables, Interfaces, Macros, Structs, Enums, etc.:
one UNWIND $batch query per label (replaces N individual MERGEs)
- Parameters: one UNWIND batch for all function params in the file
- Class→Function CONTAINS: one UNWIND batch
- Nested Function CONTAINS: one UNWIND batch
- Imports (JS): one UNWIND batch
- Imports (other): one UNWIND batch
- Ruby modules: one UNWIND batch
- Ruby module_inclusions: one UNWIND batch
Also removed:
- Stray info_logger("Executing add_file_to_graph with my change!") on every file
- info_logger("Processing import: ...") on every import statement
- Pre-seeded repo_path_str parameter: callers can supply the resolved repo path
string to skip a DB round-trip (MATCH Repository) per file — used by the
build_graph_from_path_async loop which knows the repo path upfront.
- asyncio.sleep(0.01) per file → sleep(0) every 50 files.
Old: 28,600 × 10ms = ~286s (~5min) of pure dead wait for nucleus.
New: ~0s overhead (yields to event loop only when needed).
Smoke-tested against local neo4j-cgc (admin123) — all node types and
relationship types verified correct after the rewrite.
CONTAINS* chain delete only removes nodes reachable from the Repository node. Nodes orphaned by a previous crash or partial index survive and retain their old CALLS/INHERITS relationships. When the next --force reindex runs MERGE to recreate nodes, it matches these orphaned nodes and stacks new CALLS on top of the old ones, producing duplicates (755k → 1.05M on nucleus). Phase 2 batches a path-prefix MATCH+DETACH DELETE (5000 nodes at a time) to catch any orphaned nodes not in the CONTAINS chain, ensuring the graph is truly empty before the new index is written.
DETACH DELETE on nodes with many relationships causes Neo4j to exceed transaction memory limits (11.2 GiB on remote: Neo.TransientError.General.MemoryPoolOutOfMemoryError). Fix: in Phase 2, delete CALLS/INHERITS/CONTAINS relationships in batches of 10000 first (both outbound and inbound), then delete the now-bare nodes in batches of 5000 using plain DELETE (no cascade needed).
|
@kbhaskarla is attempting to deploy a commit to the shashankss1205's projects Team on Vercel. A member of the Team first needs to authorize it. |
ModQA
added a commit
to ModQA/CodeGraphContext
that referenced
this pull request
Jun 9, 2026
…ction_call_groups MERGE on Neo4j relationships acquires an ExclusiveLock on both endpoint nodes per edge to check for existing relationships. With no index on relationship properties this costs ~130ms/edge, making large repos (35k+ CALLS edges) take 60-90 minutes for the post-processing phase. Every call path that reaches write_function_call_groups first deletes existing CALLS edges for the affected scope: - full reindex (--force): delete_repository_from_graph clears the repo - incremental watch update: delete_outgoing_calls_from_files clears callers With no existing edges to guard against, MERGE's lock overhead is pure waste. Switching to CREATE drops write time from ~130ms/edge to ~1-2ms/edge (restoring the speedup originally introduced in PR CodeGraphContext#796, which was lost when graph_builder.py was refactored into writer.py). Non-Neo4j backends (KuzuDB, FalkorDB) retain MERGE: KuzuDB's UNWIND fallback path re-executes rows individually, making CREATE unsafe there. Closes CodeGraphContext#1159 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
macunha1
pushed a commit
to macunha1/CodeGraphContext
that referenced
this pull request
Jun 21, 2026
…r-improvements perf/fix: V3 CALLS batching, watcher fixes, incremental rebuild, delete_repository robustness
macunha1
pushed a commit
to macunha1/CodeGraphContext
that referenced
this pull request
Jun 21, 2026
…ction_call_groups MERGE on Neo4j relationships acquires an ExclusiveLock on both endpoint nodes per edge to check for existing relationships. With no index on relationship properties this costs ~130ms/edge, making large repos (35k+ CALLS edges) take 60-90 minutes for the post-processing phase. Every call path that reaches write_function_call_groups first deletes existing CALLS edges for the affected scope: - full reindex (--force): delete_repository_from_graph clears the repo - incremental watch update: delete_outgoing_calls_from_files clears callers With no existing edges to guard against, MERGE's lock overhead is pure waste. Switching to CREATE drops write time from ~130ms/edge to ~1-2ms/edge (restoring the speedup originally introduced in PR CodeGraphContext#796, which was lost when graph_builder.py was refactored into writer.py). Non-Neo4j backends (KuzuDB, FalkorDB) retain MERGE: KuzuDB's UNWIND fallback path re-executes rows individually, making CREATE unsafe there. Closes CodeGraphContext#1159 Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Performance and correctness fixes for large-repo indexing, watcher reliability, and
delete_repositoryrobustness. All changes battle-tested on a 28K-file Java codebase.Changes
Performance
V3 CALLS creation (
graph_builder.py)_resolve_function_callhelper + V3_create_all_function_callswith UNWIND batchingBatched
add_file_to_graph(graph_builder.py)Incremental CALLS/INHERITS rebuild on watch events (
graph_builder.py,watcher.py)Bug Fixes
Memory exhaustion on large repos (
watcher.py)all_file_data.clear()after linking pass prevents unbounded memory growth in watch modeDuplicate CALLS accumulation on watch events (
graph_builder.py,watcher.py)Variable-length CONTAINS path in index check (
cli_helpers.py)[:CONTAINS]->[:CONTAINS*]inindex_helperto correctly traverse nested directory hierarchydelete_repository_from_graphcorrectness (graph_builder.py)watch_helperfallback indexed check (cli_helpers.py)Testing
Validated on 28,663-file Java codebase (256K functions, heavy Spring/Apache/JUnit dependencies):