Skip to content

Refactor Neo4j adapter ingestion loops to use managed transactions - #1171

Merged
Shashankss1205 merged 1 commit into
CodeGraphContext:mainfrom
roshankumar0036singh:feature/neo4j-transaction-refactor
Jun 10, 2026
Merged

Shashankss1205 merged 1 commit into
CodeGraphContext:mainfrom
roshankumar0036singh:feature/neo4j-transaction-refactor

Conversation

@roshankumar0036singh

@roshankumar0036singh roshankumar0036singh commented Jun 10, 2026 •

Copy link
Copy Markdown
Contributor

This PR introduces a comprehensive architectural refactor of the database ingestion pipeline for Neo4j. Previously, the ingestion methods relied on basic begin_transaction() contexts which lacked automatic retry semantics on transient database errors. Additionally, multi-batch data insertion methods executed individual transactions per batch, leaving severe atomicity gaps where a mid-run crash would result in partial writes and orphaned nodes.

closes #1101

This update resolves these architectural flaws by migrating the adapter to utilize natively managed execution wrappers (session.execute_write).

Extracted backend resolution logic into a centralized get_backend_type utility.
Introduced execute_write_operation and execute_read_operation to intelligently wrap Neo4j writes with auto-retry support and manage fallback logic for legacy and non-Neo4j drivers (e.g. KuzuDB, FalkorDB).
Atomic Method-Level Closures (writer.py & post_resolution.py)

Refactored over 20+ ingestion loops, including complex multi-stage methods (write_datasource_graph, write_orm_mappings, write_maven_build_graph, etc.) to securely group all batches into a single def _work(session): closure block.
This ensures that if any part of the method fails or is interrupted, the transaction rolls back cleanly, effectively eliminating the orphaned node issue.

Resolved associated runtime issues like proper loop variable closure binding (_i=i) where needed, and local variable capturing (NameError fixes for variables like cnt and deleted).
Read-Only Safeties

Properly designated read queries like CALL db.labels() and graph re-resolution queries in post_resolution.py to use execute_read_operation to avoid unnecessary write lock acquisitions.

Guarded against transient retry mutation risks by moving outer-scope mutations (like dict updates) entirely outside of the retryable read/write blocks.

Added comprehensive unit tests in tests/unit/tools/test_graph_writer_transactions.py to continuously assert the correct driver routing and retry behavior across the neo4j (modern and legacy) and fallback code paths.
image

- Group writes belonging to one logical operation under single transactions
- Auto-retry transient errors via execute_write_operation closures
- Maintain legacy session.write_transaction support
- Clean rollback semantics on interrupted execution
@vercel

vercel Bot commented Jun 10, 2026

Copy link
Copy Markdown

@roshankumar0036singh is attempting to deploy a commit to the shashankss1205's projects Team on Vercel.

A member of the Team first needs to authorize it.

@roshankumar0036singh

Copy link
Copy Markdown
Contributor Author

@Shashankss1205 can you review this

@Shashankss1205
Shashankss1205 merged commit f508cdc into CodeGraphContext:main Jun 10, 2026
10 of 12 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog tasks to Done in CGC Progress Board Jun 10, 2026
ModQA added a commit to ModQA/CodeGraphContext that referenced this pull request Jun 11, 2026
…tion scope, and MATCH optimisation

Three compounding issues caused slow CALLS edge writes on Neo4j.

1. Broken backend detection in write_function_call_groups: the legacy
   getattr chain returned the method object (always truthy) instead of
   calling it, silently using MERGE instead of CREATE on neo4j/nornic
   backends — negating PR CodeGraphContext#1161 entirely. Replaced with the canonical
   get_backend_type() call already used elsewhere in the file.

2. Single managed transaction across all CALLS batches (introduced by
   PR CodeGraphContext#1171): execute_write wrapped the entire _work closure — all 100k+
   edges in one transaction. Neo4j held write locks and accumulated WAL
   entries until the final commit, producing 46-minute silent gaps with
   ~167ms/edge observed in production logs. Replaced with a plain session
   so each 1000-edge batch gets its own execute_write — per-batch
   auto-retry and clean rollback without the lock contention.

3. Inefficient MATCH on called nodes: the called MATCH used only
   name+path (2 of 3 composite index properties), causing Neo4j to scan
   all matching nodes and filter line_number in a WHERE clause — 26,680
   DbHits per lookup vs 2 DbHits with the full composite index. Split
   each batch into fast-path (called_line_number known: full 3-property
   composite index hit) and slow-path (ambiguous resolution: 2-property
   prefix scan, correct behaviour preserved).

   Same fix applied to post_resolution.py where the caller MATCH had no
   label (full node scan, no index possible) and the called MATCH had the
   same 2-property issue.

4. Fixed NameError in graph_builder.py where cgcignore_path was
   referenced in _build_graph_from_scip but never passed as a parameter,
   causing SCIP indexing to always fall back to Tree-sitter.

Benchmarks (~97k fn→fn CALLS edges, Neo4j backend):
  Before (MERGE, single tx):           ~138ms/edge, ~1697s total
  After, 1 repo  (~13k functions):       3.3ms/edge,   320s  (43x faster)
  After, 2 repos (~27k functions, CI):   7.1ms/edge,   689s  (19x faster)

Fast/slow path split: 28% fast (known called_line_number, 2 DbHits),
71% slow (ambiguous resolution, index prefix scan). The fast path
delivers 13,340x fewer DbHits per row — proportionally larger benefit
on graphs with hundreds of thousands of nodes.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
macunha1 pushed a commit to macunha1/CodeGraphContext that referenced this pull request Jun 21, 2026
Integrate execute_write/read_operation wrappers across GraphWriter while
preserving legacy Windows delete fallback, INCLUDES rel cleanup, and
dangling Module purge from main.

Co-authored-by: Cursor <[email protected]>
macunha1 pushed a commit to macunha1/CodeGraphContext that referenced this pull request Jun 21, 2026
…tion scope, and MATCH optimisation

Three compounding issues caused slow CALLS edge writes on Neo4j.

1. Broken backend detection in write_function_call_groups: the legacy
   getattr chain returned the method object (always truthy) instead of
   calling it, silently using MERGE instead of CREATE on neo4j/nornic
   backends — negating PR CodeGraphContext#1161 entirely. Replaced with the canonical
   get_backend_type() call already used elsewhere in the file.

2. Single managed transaction across all CALLS batches (introduced by
   PR CodeGraphContext#1171): execute_write wrapped the entire _work closure — all 100k+
   edges in one transaction. Neo4j held write locks and accumulated WAL
   entries until the final commit, producing 46-minute silent gaps with
   ~167ms/edge observed in production logs. Replaced with a plain session
   so each 1000-edge batch gets its own execute_write — per-batch
   auto-retry and clean rollback without the lock contention.

3. Inefficient MATCH on called nodes: the called MATCH used only
   name+path (2 of 3 composite index properties), causing Neo4j to scan
   all matching nodes and filter line_number in a WHERE clause — 26,680
   DbHits per lookup vs 2 DbHits with the full composite index. Split
   each batch into fast-path (called_line_number known: full 3-property
   composite index hit) and slow-path (ambiguous resolution: 2-property
   prefix scan, correct behaviour preserved).

   Same fix applied to post_resolution.py where the caller MATCH had no
   label (full node scan, no index possible) and the called MATCH had the
   same 2-property issue.

4. Fixed NameError in graph_builder.py where cgcignore_path was
   referenced in _build_graph_from_scip but never passed as a parameter,
   causing SCIP indexing to always fall back to Tree-sitter.

Benchmarks (~97k fn→fn CALLS edges, Neo4j backend):
  Before (MERGE, single tx):           ~138ms/edge, ~1697s total
  After, 1 repo  (~13k functions):       3.3ms/edge,   320s  (43x faster)
  After, 2 repos (~27k functions, CI):   7.1ms/edge,   689s  (19x faster)

Fast/slow path split: 28% fast (known called_line_number, 2 DbHits),
71% slow (ambiguous resolution, index prefix scan). The fast path
delivers 13,340x fewer DbHits per row — proportionally larger benefit
on graphs with hundreds of thousands of nodes.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
@Shashankss1205 Shashankss1205 added gssoc:approved GSSoC validation: counts toward scoring level:advanced GSSoC difficulty: 55 pts contributor / 30 mentor mentor:Shashankss1205 GSSoC mentor attribution: credits reviewing mentor quality:exceptional GSSoC quality: x1.5 contributor / +10 mentor type:refactor GSSoC type bonus: refactor labels Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

gssoc:approved GSSoC validation: counts toward scoring level:advanced GSSoC difficulty: 55 pts contributor / 30 mentor mentor:Shashankss1205 GSSoC mentor attribution: credits reviewing mentor quality:exceptional GSSoC quality: x1.5 contributor / +10 mentor type:refactor GSSoC type bonus: refactor

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

DB: Wrap Neo4j Driver Operations in Explicit Write Transactions

2 participants