Skip to content

fix(indexer): self-heal missing File->CONTAINS edges caused by empty-index race - #1722

Merged
Shashankss1205 merged 1 commit into
CodeGraphContext:mainfrom
MorningStar0709:fix/contains-invariant-repair
Sep 24, 2026
Merged

fix(indexer): self-heal missing File->CONTAINS edges caused by empty-index race#1722
Shashankss1205 merged 1 commit into
CodeGraphContext:mainfrom
MorningStar0709:fix/contains-invariant-repair

Conversation

@MorningStar0709

@MorningStar0709 MorningStar0709 commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #1721 — full root-cause analysis and before/after evidence in the issue.

On FalkorDB Lite, a property index created while the parallel write workers are
still running can come up operational but empty: it participates in query
routing but matches zero rows, while a label scan finds all values. Since
#1393, the per-file MATCH (f:File) MATCH (n:<Label> {… occurrence_index …}) MERGE (f)-[:CONTAINS]->(n) linking statement keys part of its MATCH on
occurrence_index, so under this race the statement silently writes zero
File-[:CONTAINS] edges — nodes exist, no edges, no error.

Observed in production indexing of a 419-file Python repo (CGC 0.6.13):

  • cgc stats <repo> → Functions 0 / Classes 0 (the scoped counters traverse
    Repository-[:CONTAINS*]->Function);
  • MATCH (f:File)-[:CONTAINS]->(n:Function) RETURN count(n) → 0, while a
    global MATCH (n:Function) → 3,589;
  • the same equality predicate forced off-index (toInteger(n.occurrence_index) = 0)
    → 3,589. Index routing was the failure, not the data.

Changes

  1. writer.py — GraphWriter.repair_missing_contains_links(repo_path_str)
    Post-index invariant check: for Function / Class / Variable, if the
    repo prefix has symbol nodes but zero File-[:CONTAINS] edges, log a
    warning and back-fill the edges from the already-stored path property
    (MATCH (f:File) WHERE f.path STARTS WITH $repo_path MATCH (n:<Label>) WHERE n.path = f.path MERGE (f)-[:CONTAINS]->(n)).
    • Idempotent: MERGE on an already-linked pair is a no-op.
    • Conservative: partially-linked labels are left alone (zero-vs-nonzero
      distinguishes this class of total failure from per-file defects).
    • Backends: uses the existing execute_write_operation helper, so it is
      safe on Neo4j / Nornic managed transactions and passthrough on
      FalkorDB / KuzuDB.
  2. pipeline.py — call the check after the C++ out-of-line linking pass
    at the end of post-processing, with job status + warning surfaces.

Out of scope (intentionally): the underlying empty-index race lives in
FalkorDB Lite index building; this PR detects and repairs its only observable
consequence inside CGC rather than patching third-party internals. A separate
upstream report is recommended.

Test plan

New tests/unit/tools/test_contains_invariant_repair.py (6 tests, following
the _RecordingSession/_FakeDBManager pattern from
test_graph_builder_perf_fixes.py):

  • back-fill runs once per violating label; return value reports counts
  • back-fill query is path-scoped (STARTS WITH $repo_path) and joins
    n.path = f.path pairwise
  • healthy graph (all labels fully linked) issues no writes
  • empty repo is a no-op
  • partially-linked label is left alone
  • pipeline.py wires the check in

Verified locally:

  • TDD red→green: all 6 failed with AttributeError before the implementation.
  • Mutation check: disabling the guard clause fails 2 tests.
  • pytest tests/unit → 1,484 passed, 8 skipped, 9 pre-existing failures
    byte-identical to baseline (verified via git stash; scip/sse env issues,
    unrelated).
  • ruff on changed files: no new findings.

Real-world evidence: the exact back-fill statement this PR adds recovered
3,589 Function / 250 Class / 15,529 Variable File-[:CONTAINS] edges on a
live corrupt index, after which cgc stats <repo> reported correct counts.

@MorningStar0709
fix(indexer): self-heal missing File->CONTAINS edges caused by empty-…
2dc7348 
…index race

On FalkorDB Lite an index created while parallel write workers are still
running can come up operational but empty: it routes queries and matches
zero rows while a label scan finds all values. Since CodeGraphContext#1393 the per-file
linking MERGE keys its MATCH partly on occurrence_index, so under this
race it silently writes zero File-[:CONTAINS] edges (nodes exist, no
edges, no error) and scoped counters that traverse
Repository-[:CONTAINS*]->Function report 0 functions/classes.

Add GraphWriter.repair_missing_contains_links(repo_path): a post-index
invariant check that, for each of Function/Class/Variable, warns and
back-fills the edges from the stored path property when the label has
nodes under the repo prefix but zero File-[:CONTAINS] links. Idempotent
(MERGE), conservative (partially-linked labels are left alone), and
backend-agnostic via execute_write_operation. Wire it into the pipeline
after the C++ linking pass.

Tests: tests/unit/tools/test_contains_invariant_repair.py (6 cases,
RecordingSession pattern). Mutation-verified guard coverage.
@github-project-automation github-project-automation Bot moved this to Backlog tasks in CGC Progress Board Sep 20, 2026
@vercel

vercel Bot commented Sep 20, 2026

Copy link
Copy Markdown

@MorningStar0709 is attempting to deploy a commit to the shashankss1205's projects Team on Vercel.

A member of the Team first needs to authorize it.

@MorningStar0709

MorningStar0709 commented Sep 20, 2026

Copy link
Copy Markdown
Contributor Author

Heads-up on the red check, so it does not read as a defect in this change:

End-to-end Tests -> test (windows-latest) fails at collection, before anything this PR touches:

RuntimeError: Could not find lbug C API shared library.
  Set LBUG_C_API_LIB_PATH or download a shared lib
  (e.g. run LBUG_LIB_KIND=shared bash scripts/download_lbug.sh)
FAILED tests/e2e/test_verify_databases_parity.py::test_database_parity_e2e
  - RuntimeError: Indexing process failed for ladybugdb with exit code 1
1 failed, 2 skipped in 0.58s

Two observations:

  1. The End-to-end Tests workflow is also red on every recent main push (last 10, including the 0.6.13 release bump), so this looks pre-existing rather than PR-induced.
  2. Build Test -> test (windows-latest) is green here, which is consistent with the gap being specific to how the e2e job provisions the ladybug shared library on Windows.

Build Test, Database Parity Check, Lint and CodeQL are all green on this branch. Happy to rebase or adapt if a re-run is preferred.

(Separately: #1725 is unrelated to this race — it only touches cli/main.py output streams.)

Shashankss1205
Shashankss1205 approved these changes Sep 24, 2026
View reviewed changes

@Shashankss1205 Shashankss1205 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strong work — the investigation in #1721 is what makes this mergeable. Distinguishing "index routing failed" from "data missing" by forcing the predicate off-index with toInteger(n.occurrence_index) = 0 and getting 3,589 back is a clean, decisive experiment, and it's what turns this from a guess into a diagnosis.

The repair is scoped the way I'd want:

  • Zero-vs-nonzero is the right trigger. Backfilling only when a label has symbol nodes and zero CONTAINS edges separates this class of total failure from ordinary per-file defects, so a partially-linked repo is left alone rather than papered over. A repair routine that fires on partial data is how you turn a visible bug into an invisible one.
  • Backfilling from the stored path property rather than re-deriving it means the repair uses data the failed statement already wrote successfully — no re-parse, no new source of truth.
  • MERGE keeps it idempotent, so a re-run after a successful index is a no-op rather than a duplicate-edge generator.
  • Routing through execute_write_operation keeps it correct on Neo4j/Nornic managed transactions and passthrough elsewhere.

I also agree with the scope call: the empty-index race lives in FalkorDB Lite's index building, and patching third-party internals from here would be worse than detecting the consequence. Please do file that upstream report — link it back on #1721 so the trail survives.

Verified locally against the full unit + integration suite: 1596 passed, 8 skipped.

@Shashankss1205
Shashankss1205 merged commit cdcdc3f into CodeGraphContext:main Sep 24, 2026
19 of 21 checks passed
@Shashankss1205 Shashankss1205 added gssoc:approved GSSoC validation: counts toward scoring level:advanced GSSoC difficulty: 55 pts contributor / 30 mentor mentor:Shashankss1205 GSSoC mentor attribution: credits reviewing mentor quality:exceptional GSSoC quality: x1.5 contributor / +10 mentor type:bug GSSoC type bonus: bug labels Sep 24, 2026
This was referenced Sep 24, 2026
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Shashankss1205 Shashankss1205 Shashankss1205 approved these changes

Assignees

No one assigned

Labels

gssoc:approved GSSoC validation: counts toward scoring level:advanced GSSoC difficulty: 55 pts contributor / 30 mentor mentor:Shashankss1205 GSSoC mentor attribution: credits reviewing mentor quality:exceptional GSSoC quality: x1.5 contributor / +10 mentor type:bug GSSoC type bonus: bug

Projects

CGC Progress Board
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

bug(indexer/falkordb): empty-index race during parallel writes silently skips all File-[:CONTAINS] linking; scoped cgc stats reports 0 functions/classes

2 participants

@MorningStar0709 @Shashankss1205