gateway: indexing blocks the event loop — /health and all routes stop responding during grammar download and import pre-scan #1745

Summary

Once jobs run (see #1744), indexing a repository whose grammars are not cached makes the
whole gateway unresponsive, including /health. Without internet access every download attempt
blocks for 60 s (tree-sitter-language-pack's HTTP timeout), and CGC retries it for every file, so
a liveness probe kills the process. With grammars present, the synchronous phases still block the
loop for seconds per repository.

Cause

• run_tree_sitter_index_async calls pre_scan_for_imports(...) synchronously on the loop
  (tools/indexing/pipeline.py L124; also scip_pipeline.py L110 and L285). The pre-scan builds
  parsers via get_parser → TreeSitterParser → get_language_safe →
  tree_sitter_language_pack.get_language, which downloads the platform bundle from GitHub on
  a cache miss.
• The first parser init always calls get_language('python') (utils/tree_sitter_manager.py L58),
  whatever language the repo uses.
• A failed parser init is not cached (get_parser, graph_builder.py L142-146), so the download is
  retried for each pre-scan extension and then for each file. Locally, with a silent endpoint, a
  1-file repo froze the loop for 120 s and a 3-file repo for 240 s. Both jobs ended completed
  with no definitions.
• Moving the pre-scan to asyncio.to_thread is not enough on its own:
  tree-sitter-language-pack's Python get_language does its network I/O without releasing the GIL
  (see bug: Python get_language() holds the GIL during on-demand download xberg-io/tree-sitter-language-pack#203). parse_file, which already runs in a thread, froze the loop for 60 s per
  call in the same test.
• Even with grammars present, discovery, the import pre-scan and all post-processing/graph writes
  (pipeline.py L271-420) run on the loop. The pre-scan alone took 3–6 s for a repo with 371
  Python files.

Environment

codegraphcontext 0.6.13 (same on main), tree-sitter-language-pack 1.20.0, Python 3.12, Linux
container without internet egress.

Suggestions

1. Run the pre-scan (and ideally the post-processing phases) via asyncio.to_thread.
2. Before the pre-scan, call tree_sitter_language_pack.prefetch(langs) in a
   thread. It releases the GIL: in the same test the loop's largest gap was 0.05 s. Remember a
   failed grammar so it is not retried for every file.
3. Offline deployments: document fetching grammars at build time with prefetch([...]) or the
   release bundle (download([...]) is currently a no-op, see bug: download([...]) is a no-op for known languages and returns the requested count xberg-io/tree-sitter-language-pack#201), and a
   TREE_SITTER_LANGUAGE_PACK_MANIFEST_URL that refuses connections (e.g. a closed local port)
   so a missing grammar fails immediately. A blackholed address still waits 60 s. A cached
   manifest.json overrides MANIFEST_URL.

Disclosure: the investigation and this report were prepared with AI assistance (Claude Code) and reviewed by a human before posting.