Repository navigation
ci: Database Parity Check flaky on REL_CALLS (spread 2 vs allowed 1) #1440
Description
Database Parity Check is the last consistently-red workflow on main. It fails on one metric only.
The failure
Metric | KuzuDB | LadybugDB | FalkorDB | Neo4j | Match?
NODE_Class | 243 | 243 | 243 | 243 | YES
NODE_Function | 1551 | 1551 | 1551 | 1551 | YES
NODE_Variable | 1462 | 1462 | 1462 | 1462 | YES
...all node metrics match exactly...
REL_CALLS | 647 | 647 | 649 | 648 | NO <-- spread 2
REL_CONTAINS | 4505 | 4505 | 4505 | 4505 | YES
REL_HAS_PARAMETER | 1230 | 1230 | 1230 | 1230 | YES
REL_IMPORTS | 249 | 249 | 249 | 249 | YES
REL_INHERITS | 141 | 141 | 141 | 141 | YES
allowed_spread = {"REL_IMPORTS": 6, "REL_CALLS": 1} — the observed spread is 2, so it fails by one edge over an already-accepted tolerance. That is 0.3 % of ~648 edges, and every other metric is exact.
It is flaky, not broken
Over the last 100 runs of this workflow: 32 success / 58 failure / 10 action_required. Two consecutive runs on the same main commit gave different orderings:
| run | KuzuDB | LadybugDB | FalkorDB | Neo4j |
|---|---|---|---|---|
CI de5a66c |
647 | 647 | 649 | 648 |
CI (earlier, 567e73c, pre-dating the recent fix batch) |
647 | 648 | 649 | 648 |
| my local run | 648 | 648 | 647 | 649 |
The same backend lands on different numbers in different runs, so this is not a fixed per-backend mapping difference.
Not caused by the recent audit-fix batch
Reproduced identically at 567e73c, before those merges landed — same metric, same magnitude.
What I could not reproduce locally
I tried fairly hard to pin this down and failed, so recording the negative results:
- Repeat runs, same backend: KùzuDB indexed the fixture 4× →
647every time. - Cross-backend, locally: dumped the full CALLS edge set (caller/callee name, path, line,
full_call_name,confidence_label) from KùzuDB and FalkorDB and diffed them — 647 each, byte-identical edge sets, zero edges unique to either. - CPU contention: GitHub runners are 2-core and the indexer uses a hardcoded
asyncio.Semaphore(10)overasyncio.to_thread, so I re-ran pinned withtaskset -c 0,13× →647every time, identical edge sets. - Dependency drift: CI and my environment resolve identical versions —
tree-sitter 0.25.2,tree-sitter-language-pack 1.13.3,tree-sitter-c-sharp 0.23.5,kuzu 0.11.3,falkordblite 0.9.0,falkordb 1.2.0,ladybug 0.18.3. (falkordbis unpinned at>=1.0,<1.6, but both resolved to 1.2.0.)
So: same code, same dependencies, same fixture, deterministic locally, divergent on the runner.
Where I would look next
writer.pyusescalls_keyword = "CREATE" if backend in ("neo4j", "nornic") else "MERGE". Neo4j can therefore hold duplicates that MERGE-backends collapse — worth having the test reportcount(r)andcount(DISTINCT ...)per backend to tell "extra edge" from "duplicate edge" apart.- The existing tolerance comment attributes drift to KùzuDB/LadybugDB dropping ≤1 edge when the fast/slow MATCH split cannot bind
called_line_number. If that is the mechanism, the drop is now sometimes 2 and the tolerance is simply stale.
Deciding what to do
Two options, both defensible:
- Raise
allowed_spread["REL_CALLS"]to 2. Turns a flaky red green immediately. Cost: a real regression that adds or drops 2 CALLS edges would pass silently. - Leave it red and fix the underlying divergence. Correct, but needs a CI-side diagnostic (the distinct-vs-total counts above) since it does not reproduce locally.
I have deliberately not bumped the tolerance — masking a correctness divergence is a maintainer call, not something to slip into a CI-green PR.
Activity
github-actions commentedon Jul 26, 2026
Hi! 👋 Join our CodeGraphContext Discord channel to collaborate: https://discord.gg/dR4QY32uYQ
Shashankss1205 commentedon Jul 26, 2026
Diagnostic landed in #1441 — the Actions log now prints the per-backend CALLS edge sets and the diff, so this no longer needs local reproduction.
It already narrows the spread of 2 to exactly two edges, and both are wrong regardless of backend:
That reframes the issue: this is not a backend-parity problem, it is two bad edges that backends happen to disagree about because they are unresolved.
languages_are_compatibleshould have blocked it.nameis a filename and whoseline_numberis null — exactly the shape the existing tolerance comment blames ("cannot bind an exactcalled_line_number").So the fix is to stop emitting them, not to widen
allowed_spread. Option 1 from the original description should be dropped.Also surfaced, unrelated to parity: 11 duplicate CALLS edges on every backend (648 total vs 637 distinct, keyed caller→callee@call_line). Identical across all four, so it doesn't affect the assertion, but worth confirming those are genuinely multiple calls on one line rather than double-writes.
sahilmathur254 commentedon Jul 26, 2026
The diagnostics in #1441 make the first fix boundary fairly clear. I can take a focused PR that prevents the cross-project Java-to-Kotlin match and rejects file-shaped TypeScript callees without a concrete line, with regression fixtures for both cases. I would leave the 11 identical duplicate edges across all backends for a separate investigation.
Would you like me to take that implementation, or are you already continuing it from the diagnostic work?
Shashankss1205 commentedon Aug 1, 2026
Closing as resolved — the Database Parity Check has been green since
11a86d4.The REL_CALLS spread was nondeterminism in CALLS resolution, fixed by
11a86d4"fix(calls): make CALLS resolution deterministic and stop dropping File-target edges (#1445)".The failure streak ends at
8616ba7(2026-07-26) and every run since is green. Reopen if the spread comes back.