perf(kuzudb): write_function_call_groups takes 5.8 min for 886 CALLS edges on 80-file project

Summary

When indexing a medium-sized Python project (80 files, ~600 functions) with the KuzuDB backend, write_function_call_groups takes ~5.8 minutes and consumes ~2.2 GB RAM to write 886 CALLS edges. The post-processing phase blocks the asyncio event loop because session.run() is called synchronously inside an async function, freezing the progress bar UI.

Additionally, repeated --force re-indexing causes the KuzuDB database file to grow without reclamation (observed 239 MB → 66 MB after manual rebuild for the same project, a 3.6× bloat).

Environment

CGC version: 0.4.17
OS: Windows 11 25H2
Database backend: KuzuDB (default, embedded)
Python: 3.11.15
Target project: Python package with 80 indexable files, 599 functions, 2853 variables

Reproduction

# Ensure clean database
rm ~/.codegraphcontext/global/db/kuzudb   # or delete manually on Windows

# Index a medium-sized Python project
cgc index /path/to/project --force

Observed Behavior

1. `write_function_call_groups` takes ~5.8 minutes

From the evidence log:

Wall time:          346.1s
Peak child memory:  2267.1 MB
Peak child CPU:     406.5s
DB size after:      65.64 MB
CALLS edges written: 886

The UNWIND $batch AS row MATCH ... MERGE ... Cypher query in write_function_call_groups (writer.py L669-708) dominates the entire indexing time. File parsing completes in ~30 seconds; the remaining 5+ minutes are spent writing CALLS edges.

2. Progress bar frozen during post-processing

The post-processing steps (build_inheritance_and_csharp_files, build_function_call_groups, write_function_call_groups, etc.) in pipeline.py are synchronous calls within an async function. This blocks the asyncio event loop, preventing the progress bar polling loop from executing. The UI appears stuck at "Indexing files... 99%" for the entire post-processing duration.

3. Database file bloat (3.6×)

After multiple --force re-indexing cycles across different projects, the KuzuDB database file grew to 239 MB while containing only 3 repositories and 85 files. After deleting and rebuilding from scratch, the same project produced a 66 MB database. KuzuDB does not appear to reclaim space after node/edge deletions.

Expected Behavior

CALLS edge writing should complete in seconds, not minutes, for a project of this size.
Progress bar should remain responsive during post-processing, showing a "Post-processing..." indicator instead of the frozen "Indexing files... 99%".
Database size should remain proportional to the actual data, not accumulate dead space from deletions.

Suggested Fixes

Wrap KuzuDB writes in asyncio.to_thread() — The synchronous session.run() calls in pipeline.py (lines 111-124) and writer.py block the event loop. Wrapping them would keep the UI responsive.
Optimize the MERGE query pattern — The current UNWIND + MATCH + MERGE with multi-property matching may not use KuzuDB's indexes effectively. Consider:
- Using CREATE instead of MERGE for CALLS edges (they are idempotent per indexing run)
- Reducing the number of WHERE clause conditions
- Batching into smaller chunks
Add database compaction — After --force re-indexing (which deletes and recreates data), trigger a KuzuDB checkpoint/vacuum or recreate the database file to reclaim space.

Evidence

Full evidence log attached: cgc_evidence_mineru-eval.log

Key database statistics after indexing:

Node Type	Count
Variable	2,853
Function	599
Parameter	546
Module	103
File	80
Class	26
CALLS edges	886

Hi! 👋 Join our CodeGraphContext Discord channel to collaborate: https://discord.gg/dR4QY32uYQ

[01:55:50.107] ======================================================================
[01:55:50.107] CGC KuzuDB Performance Evidence Collection
[01:55:50.107] ======================================================================
[01:55:50.107] CGC binary: D:\WorkPlace\AI\SciMonitor.venv\Scripts\cgc.exe
[01:55:50.107] Target: D:\WorkPlace\AI\SciMonitor\packages\mineru-eval
[01:55:50.107] DB path: C:\Users\skyler.codegraphcontext\global\db\kuzudb
[01:55:50.107] DB size before: 0.00 MB
[01:55:50.107]
[01:55:50.107] Command: D:\WorkPlace\AI\SciMonitor.venv\Scripts\cgc.exe index D:\WorkPlace\AI\SciMonitor\packages\mineru-eval --force
[01:55:50.107] Start time: 2026-06-10T01:55:50.107854
[01:55:50.107]
[OUTPUT] Loaded configuration from: C:\Users\skyler.codegraphcontext.env
[OUTPUT] Using database: kuzudb (source: C:\Users\skyler.codegraphcontext.env)
[OUTPUT] Force re-indexing (--force flag detected)
[OUTPUT] Resolving context...
[OUTPUT] Initializing services and database connection...
[OUTPUT] Services initialized.
[OUTPUT] Re-indexing: D:\WorkPlace\AI\SciMonitor\packages\mineru-eval
[OUTPUT]
[SNAPSHOT @341s] [{"Id":7176,"ProcessName":"cgc","CPU":0.0625,"MemMB":10.8},{"Id":6032,"ProcessName":"python","CPU":406.515625,"MemMB":2267.1},{"Id":9696,"ProcessName":"python","CPU":0.078125,"MemMB":10.9},{"Id":20344,"ProcessName":"python","CPU":0.03125,"MemMB":10.8}]
[OUTPUT] Skipped 16 unresolved call relationship(s): ambiguous_function_target=16
[OUTPUT] ┌───────────────┬───────────────────────────┬─────────────────────────────────┐
[OUTPUT] │ Call │ Reason │ Location │
[OUTPUT] ├───────────────┼───────────────────────────┼─────────────────────────────────┤
[OUTPUT] │ discretize │ ambiguous_function_target │ D:/WorkPlace/AI/SciMonitor/pack │
[OUTPUT] │ │ │ ages/mineru-eval/src/mineru_eva │
[OUTPUT] │ │ │ l/results/judge_matrix/consolid │
[OUTPUT] │ │ │ ate_api_8judge.py:50 │
[OUTPUT] │ discretize │ ambiguous_function_target │ D:/WorkPlace/AI/SciMonitor/pack │
[OUTPUT] │ │ │ ages/mineru-eval/src/mineru_eva │
[OUTPUT] │ │ │ l/results/judge_matrix/consolid │
[OUTPUT] │ │ │ ate_api_8judge.py:51 │
[OUTPUT] │ discretize │ ambiguous_function_target │ D:/WorkPlace/AI/SciMonitor/pack │
[OUTPUT] │ │ │ ages/mineru-eval/src/mineru_eva │
[OUTPUT] │ │ │ l/results/judge_matrix/consolid │
[OUTPUT] │ │ │ ate_api_8judge.py:84 │
[OUTPUT] │ scorer.judge │ ambiguous_function_target │ D:/WorkPlace/AI/SciMonitor/pack │
[OUTPUT] │ │ │ ages/mineru-eval/src/mineru_eva │
[OUTPUT] │ │ │ l/judge.py:574 │
[OUTPUT] │ AScorer.judge │ ambiguous_function_target │ D:/WorkPlace/AI/SciMonitor/pack │
[OUTPUT] │ │ │ ages/mineru-eval/src/mineru_eva │
[OUTPUT] │ │ │ l/self_correction_test.py:106 │
[OUTPUT] └───────────────┴───────────────────────────┴─────────────────────────────────┘
[OUTPUT] Successfully re-indexed: D:\WorkPlace\AI\SciMonitor\packages\mineru-eval in
[OUTPUT] 338.85 seconds
[02:01:36.263]
[02:01:36.263] ======================================================================
[02:01:36.263] RESULTS
[02:01:36.263] ======================================================================
[02:01:36.263] Exit code: 0
[02:01:36.263] Wall time: 346.1s
[02:01:36.263] Peak child memory: 2267.1 MB
[02:01:36.263] Peak child CPU: 406.5s
[02:01:36.263] DB size after: 65.64 MB
[02:01:36.265] End time: 2026-06-10T02:01:36.265646
[02:01:36.265]
[02:01:36.265] Database node counts:
[02:01:40.118] Query: MATCH (n) RETURN labels(n)[0] as label, count(n) as cnt ORDER BY cnt DESC
[02:01:40.118] Result: Resolving context...
Initializing services and database connection...
Services initialized.
[
{
"label": "Variable",
"cnt": 2853
},
{
"label": "Function",
"cnt": 599
},
{
"label": "Parameter",
"cnt": 546
},
{
"label": "Module",
"cnt": 103
},
{
"label": "File",
"cnt": 80
},
{
"label": "Class",
"cnt": 26
},
{
"label": "Directory",
"cnt": 9
},
{
"label": "Repository",
"cnt": 1
}
]
[02:01:43.715] Query: MATCH ()-[c:CALLS]->() RETURN count(c) as calls
[02:01:43.715] Result: Resolving context...
Initializing services and database connection...
Services initialized.
[
{
"calls": 886
}
]

Update: Neo4j benchmark comparison

After switching to Neo4j Community Edition (client-server, via cgc config db neo4j), the same project (and a larger one) shows dramatically different results:

Metric	KuzuDB (embedded)	Neo4j (server)	Improvement
Files indexed	80 (subdirectory)	273 (full repo)	3.4x more
Indexing time	346.1 s	43.75 s	~27x per file
Peak memory	2,267 MB	~200 MB (CGC + Neo4j idle)	~11x less
CALLS edges	886	—	—

The ~27x per-file performance gap indicates that KuzuDB's UNWIND + MATCH + MERGE query pattern (in write_function_call_groups) has a fundamental performance bottleneck with this workload, not a configuration or usage issue.

For users experiencing slow indexing: Switch to Neo4j backend:

cgc config db neo4j
# Set in ~/.codegraphcontext/.env:
# NEO4J_URI=bolt://localhost:7687
# NEO4J_USERNAME=neo4j
# NEO4J_PASSWORD=your_password

For maintainers: The KuzuDB merge query at writer.py L669-708 remains the root cause. Consider optimizing the query pattern or investigating KuzuDB's MERGE performance characteristics.

Keeping this issue open for upstream fix evaluation.

Hi! I'd like to work on this issue as my first contribution. Could this be assigned to me?

Hi I would like to work on this issue

/assign
I would like to work on this issue

Hi, i'm a GSSoC'26 contributor. i'd like to work on this issue. can you please assign it to me

Hi, I'd like to work on this issue under GSSoC'26. I've gone through the codebase and have a clear idea of the fix. Could you please assign it to me?

Hi, I would like to work on this issue under GSSoC 2026. Could you please assign it to me?

@Shashankss1205

/assign
Hi @Shashankss1205 , I would like to contribute to this issue under GSSoC 2026. I have experience with the project's tech stack and can provide a fix promptly. Kindly assign it to me. Thank you!

Hi! I noticed this issue is currently assigned. If it becomes available again or the assignee is inactive, I'd be happy to work on it. In the meantime, I'll look for other issues to contribute. Thanks!

Maintenance review — 15 October 2026

#1766 moves blocking indexing work off the gateway loop, and #1771 targets selected node/relationship batches. This issue also reports CALLS throughput and repeated-force storage growth; those require their own benchmark and disk-size regression.

Navigation Menu

Repository navigation

perf(kuzudb): write_function_call_groups takes 5.8 min for 886 CALLS edges on 80-file project #1168

Summary

Environment

Reproduction

Observed Behavior

1. `write_function_call_groups` takes ~5.8 minutes

2. Progress bar frozen during post-processing

3. Database file bloat (3.6×)

Expected Behavior

Suggested Fixes

Evidence

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

perf(kuzudb): write_function_call_groups takes 5.8 min for 886 CALLS edges on 80-file project #1168

Description

Summary

Environment

Reproduction

Observed Behavior

1. write_function_call_groups takes ~5.8 minutes

2. Progress bar frozen during post-processing

3. Database file bloat (3.6×)

Expected Behavior

Suggested Fixes

Evidence

Activity

github-actions commented on Jun 9, 2026

MorningStar0709 commented on Jun 9, 2026

MorningStar0709 commented on Jun 10, 2026

adityamsr2606 commented on Jun 10, 2026

nandinisatheesh8a-hue commented on Jun 10, 2026

gshariish-cloud commented on Jun 11, 2026

poojaarabati commented on Jun 11, 2026

sujitsingh8 commented on Jun 12, 2026

srinithithirumaran commented on Jun 16, 2026

hetbharodiya commented on Jun 21, 2026

lokeshtheprogrammer commented on Jun 28, 2026

Shashankss1205 commented on Oct 15, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

1. `write_function_call_groups` takes ~5.8 minutes