Bug: Segfault during 'cgc index' — concurrent tree-sitter execute_query() calls across worker threads (race condition in native extension)

Describe the bug

cgc index / codegraphcontext index segfaults (Fatal Python error: Segmentation fault) non-deterministically while indexing a mixed-language repository. The crash happens inside the native Tree-sitter extension when multiple worker threads call tree_sitter_manager.execute_query() concurrently on different Query/QueryCursor objects for different languages at the same time.

The Python-level faulthandler traceback (attached below) shows three threads simultaneously inside execute_query() at the moment of the crash:

Thread A: parsing a TypeScript file (languages/typescript.py:482 _find_calls)
Thread B: parsing a Python file (languages/python.py:596 _find_variables)
Thread C (current/crashing thread): parsing an HTML file (languages/html.py:68 parse)

All three call into tree_sitter_manager.py execute_query() (lines 338/354), which creates a tree_sitter.Query + tree_sitter.QueryCursor and calls .captures(node). The tree-sitter native extension (tree_sitter_language_pack.bindings.*) is apparently not safe to call concurrently from multiple threads in this way, causing an intermittent SIGSEGV.

This reproduces with both supported embedded backends (FalkorDB Lite and KùzuDB), which rules out the database layer as the cause — it is isolated to the parsing/indexing pipeline.

Steps to reproduce

Install via pipx install codegraphcontext (clean install, confirmed reproducible on a fresh pipx venv)
Index a real-world mixed-language repository with several hundred files spanning Python, HTML, TypeScript, etc. (repro repo: ~758 files)
```
cgc index /path/to/mixed-language-repo
```
Observe the progress bar advancing, then the process dies with:
```
Falha de segmentação (imagem do núcleo gravada)
```
(Segmentation fault (core dumped))
The crash point is non-deterministic — it occurred at 57%, 94%, etc. across different runs, and with different database backends (FalkorDB Lite, then KùzuDB), always inside execute_query().

Re-running with PYTHONFAULTHANDLER=1 confirms Fatal Python error: Segmentation fault with multiple threads inside tree_sitter_manager.execute_query() concurrently.

Expected behavior

Indexing should complete without crashing, regardless of how many files/languages are processed concurrently.

Root cause (suspected)

tools/indexing/pipeline.py runs file parsing with a hardcoded concurrency limit of 10 concurrent worker threads via asyncio.to_thread(parse_file, ...):

# pipeline.py, ~line 66-67
concurrency_limit = 10
semaphore = asyncio.Semaphore(concurrency_limit)

(This is also flagged in #1340 — PARALLEL_WORKERS config is defined but never actually wired into this semaphore, so there's currently no way to reduce concurrency via config either.)

Each of these worker threads independently calls into execute_query() in utils/tree_sitter_manager.py, which constructs a tree_sitter.Query(language, query_string) + QueryCursor(query).captures(node) per call. It appears the tree-sitter native bindings (via tree-sitter-language-pack) are not safe to invoke from multiple threads concurrently in this pattern — even when operating on separate Language/Query/Node objects per thread — resulting in memory corruption and a SIGSEGV in the C extension.

This is conceptually similar to #880/#881 (KùzuDB connection race), but the affected component here is the Tree-sitter parsing layer, not the database layer.

Suggested fix

One or more of:

Serialize all execute_query() calls behind a global lock (simplest, safe, but loses parsing parallelism).
Wire PARALLEL_WORKERS into pipeline.py's concurrency_limit (per Bug: PARALLEL_WORKERS config is not wired to indexing pipeline — concurrency hardcoded at 10 #1340) and default it to 1 until thread-safety of the Tree-sitter query path is verified, or document that > 1 is unsafe.
Investigate whether tree-sitter / tree-sitter-language-pack require per-thread Parser/Query instances that are never shared/cached across threads, and audit tree_sitter_manager.py caching (get_language_safe, etc.) for anything shared unsafely across threads.

Environment

codegraphcontext version: 0.5.1 (installed via pipx install codegraphcontext, clean reinstall — bug reproduces identically on a fresh install)
Python: 3.12.3
OS: Ubuntu 24.04.4 LTS (Noble Numbat), native Linux (not WSL/Docker)
tree-sitter: 0.25.2
tree-sitter-language-pack: 0.13.0
Database backends tested: FalkorDB Lite and KùzuDB — both crash the same way

Faulthandler traceback (full)

Fatal Python error: Segmentation fault

Thread 0x000070aa757fa6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa75ffb6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa767fc6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa76ffd6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa777fe6c0 (most recent call first):
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 767 in load_context_config
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 313 in should_apply_project_dotenv
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 329 in find_local_env
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 276 in load_config
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 529 in get_config_value
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/graph_builder.py", line 1038 in parse_file
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa77fff6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa94df96c0 (most recent call first):
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/utils/tree_sitter_manager.py", line 338 in execute_query
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/languages/typescript.py", line 482 in _find_calls
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/languages/typescript.py", line 191 in parse
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/tree_sitter_parser.py", line 120 in parse
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/graph_builder.py", line 1048 in parse_file
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa955fa6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa95dfb6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa967fc6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa96ffd6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa977fe6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aa97fff6c0 (most recent call first):
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 767 in load_context_config
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 313 in should_apply_project_dotenv
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 329 in find_local_env
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 276 in load_config
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 529 in get_config_value
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/utils/debug_log.py", line 22 in _get_config_value
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/utils/debug_log.py", line 58 in debug_log
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/graph_builder.py", line 1036 in parse_file
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aab49f96c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aab51fa6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aab59fb6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aab61fc6c0 (most recent call first):
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/utils/tree_sitter_manager.py", line 338 in execute_query
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/languages/python.py", line 596 in _find_variables
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/languages/python.py", line 160 in parse
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/tree_sitter_parser.py", line 120 in parse
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/graph_builder.py", line 1041 in parse_file
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Current thread 0x000070aab69fd6c0 (most recent call first):
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/utils/tree_sitter_manager.py", line 354 in execute_query
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/languages/html.py", line 68 in parse
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/tree_sitter_parser.py", line 120 in parse
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/graph_builder.py", line 1048 in parse_file
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aab71fe6c0 (most recent call first):
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 89 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aab79ff6c0 (most recent call first):
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 267 in load_config
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/config_manager.py", line 529 in get_config_value
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/graph_builder.py", line 1038 in parse_file
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58 in run
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 92 in _worker
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aac3fff6c0 (most recent call first):
  File "/usr/lib/python3.12/threading.py", line 359 in wait
  File "/usr/lib/python3.12/threading.py", line 655 in wait
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/rich/live.py", line 35 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aac8dfe6c0 (most recent call first):
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/core/database_falkordb.py", line 317 in _drain
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aac95ff6c0 (most recent call first):
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/core/database_falkordb.py", line 317 in _drain
  File "/usr/lib/python3.12/threading.py", line 1010 in run
  File "/usr/lib/python3.12/threading.py", line 1073 in _bootstrap_inner
  File "/usr/lib/python3.12/threading.py", line 1030 in _bootstrap

Thread 0x000070aaf9707080 (most recent call first):
  File "<frozen posixpath>", line 476 in _joinrealpath
  File "<frozen posixpath>", line 432 in realpath
  File "/usr/lib/python3.12/pathlib.py", line 1242 in resolve
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/tools/indexing/pipeline.py", line 78 in process_file
  File "/usr/lib/python3.12/asyncio/events.py", line 88 in _run
  File "/usr/lib/python3.12/asyncio/base_events.py", line 1987 in _run_once
  File "/usr/lib/python3.12/asyncio/base_events.py", line 641 in run_forever
  File "/usr/lib/python3.12/asyncio/base_events.py", line 674 in run_until_complete
  File "/usr/lib/python3.12/asyncio/runners.py", line 118 in run
  File "/usr/lib/python3.12/asyncio/runners.py", line 194 in run
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/cli_helpers.py", line 314 in index_helper
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/codegraphcontext/cli/main.py", line 1318 in index
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/typer/main.py", line 1515 in wrapper
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/typer/_click/core.py", line 489 in invoke
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/typer/_click/core.py", line 746 in invoke
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/typer/core.py", line 1115 in invoke
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/typer/core.py", line 183 in _main
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/typer/core.py", line 1193 in main
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/typer/_click/core.py", line 807 in __call__
  File "/home/vfeitoza/.local/lib/python3.12/site-packages/typer/main.py", line 1137 in __call__
  File "<string>", line 5 in <module>

Extension modules: yaml._yaml, numpy._core._multiarray_umath, numpy.linalg._umath_linalg, psutil._psutil_linux, psutil._psutil_posix, _cffi_backend, markupsafe._speedups, lxml._elementpath, lxml.etree, zmq.backend.cython._zmq, tornado.speedups, msgpack._cmsgpack, tree_sitter_c_sharp._binding, tree_sitter_embedded_template._binding, tree_sitter_yaml._binding, tree_sitter_language_pack.bindings.python, tree_sitter_language_pack.bindings.typescript, tree_sitter_language_pack.bindings.html, tree_sitter_language_pack.bindings.tsx, tree_sitter_language_pack.bindings.javascript (total: 20)

Related issues

Bug: PARALLEL_WORKERS config is not wired to indexing pipeline — concurrency hardcoded at 10 #1340 — confirms PARALLEL_WORKERS is defined but not wired into pipeline.py's hardcoded concurrency_limit = 10, which currently prevents reducing concurrency as a workaround via config.
bug: KùzuDB single Connection shared across concurrent threads causes silent data corruption and non-deterministic crashes #880 / Fix Kuzu shared-connection race with query lock #881 — similar pattern of unsynchronized concurrent access causing crashes/corruption (there for the KùzuDB connection, fixed by adding a lock); this issue is the analogous problem in the Tree-sitter parsing layer.

Hi! 👋 Join our CodeGraphContext Discord channel to collaborate: https://discord.gg/dR4QY32uYQ

fix(parsers): serialise tree-sitter query execution across threads (#…

Navigation Menu

Repository navigation

Bug: Segfault during 'cgc index' — concurrent tree-sitter execute_query() calls across worker threads (race condition in native extension) #1370

Describe the bug

Steps to reproduce

Expected behavior

Root cause (suspected)

Suggested fix

Environment

Faulthandler traceback (full)

Related issues

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Bug: Segfault during 'cgc index' — concurrent tree-sitter execute_query() calls across worker threads (race condition in native extension) #1370

Description

Describe the bug

Steps to reproduce

Expected behavior

Root cause (suspected)

Suggested fix

Environment

Faulthandler traceback (full)

Related issues

Activity

github-actions commented on Jul 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions