CGC Watch continuously consumes memory until it eats all available RAM. #744

Activity

1. github-actions commented on Mar 21, 2026

   

   github-actionsbot

   on Mar 21, 2026 – with GitHub Actions

   Contributor

   More actions

   Hi! 👋 Join our CodeGraphContext Discord channel to collaborate: https://discord.gg/dR4QY32uYQ
2. Shashankss1205 commented on Apr 7, 2026

   

   Shashankss1205

   on Apr 7, 2026

   Collaborator

   More actions

   Hey @Slamik can you please help us reproduce this issue

   I would be grateful if I can get to know the repo name/size, time taken in hrs before freezing, what version and db of cgc you used?
3. Shashankss1205

   added this to CGC Progress Boardand CGC Progress Boardon Apr 29, 2026
4. github-project-automation

   moved this to Backlog in CGC Progress Boardon Apr 29, 2026
5. Gurudutta22 commented on Jun 4, 2026

   

   Gurudutta22

   on Jun 4, 2026

   More actions

   I’m interested in contributing to this project and would love to work on this issue. I would be happy to implement the required changes. Please assign me this issue.
   /assign
6. Shashankss1205

   added this to CodeGraphContext Roadmapon Jun 6, 2026
7. github-project-automation

   moved this to Todo in CodeGraphContext Roadmapon Jun 6, 2026
8. Shashankss1205

   added this to CodeGraphContext Roadmapon Jun 6, 2026
9. apklein commented on Aug 13, 2026

   

   apklein

   on Aug 13, 2026

   More actions

   @Shashankss1205 I have a deterministic reproducer and what I believe is the root cause. Measured on cgc 0.5.7 — I diffed my installed core/watcher.py against v0.5.7 on GitHub and it is byte-identical, so none of this is local drift.

   Root cause: one OS thread per filesystem event, held for the full debounce

   RepositoryEventHandler._debounce (core/watcher.py:193) creates a threading.Timer — a real OS thread — for every filesystem event, and that thread stays alive for the whole debounce_interval (2.0 s) before it decides whether the file is even relevant.

   The amplifier is on_deleted (core/watcher.py:357):

   def on_deleted(self, event):
       if not event.is_directory:
           self._debounce(event.src_path, lambda: self._handle_modification(event.src_path))

   on_created and on_modified both gate on _is_supported_code_file before _debounce, so an ignored path or an unparsed extension costs nothing. on_deleted has no gate at all — the .cgcignore check happens later, inside _handle_modification_locked (core/watcher.py:253), which is after the thread exists and after it has queued on _update_lock.

   So the thread count follows the raw delete rate, independent of .cgcignore.

   Reproducer

   Against an already-indexed repo with cgc watch running, in a directory that is already ignored (target/), using an extension that is not parsed (.tmp) — i.e. doubly excluded:

   mkdir -p target/probe
   for i in $(seq 1 800); do : > target/probe/p_$i.tmp; done
   sleep 1
   rm -f target/probe/*.tmp &
   for s in 1 2 3 4 5 6 8 10; do
     echo "threads=$(ps -M -p <WATCH_PID> | tail -n +2 | wc -l)"; sleep 0.35
   done

   Result:

   baseline threads=4
   t=2  threads=804      <- 800 delete events -> 804 live OS threads
   t=6  threads=804
   t=8  threads=660      <- draining as the 2 s timers fire
   after: threads=4
   

   All 804 threads did nothing: the path was ignored and the extension unparsed. They existed only to sleep 2 s.

   Real-world triggers are ordinary: mvn clean, flutter clean, git checkout between branches, rm -rf node_modules. On a polyglot monorepo these delete tens of thousands of files in a burst.

   Why the original report sees unbounded RAM and I do not

   This is the part I think explains @Slamik's 128 GB.

   On macOS there is a hard per-process thread ceiling:

   $ sysctl kern.num_taskthreads
   kern.num_taskthreads: 4096
   

   So the watcher saturates at ~4096 threads and stops growing. Three separate watcher processes, same box, same repo:

   pid	age	phys_footprint	peak	live threads	stack regions
   A	2 h 57 m	290 MB	354 MB	3	4101
   B	2 m	191 MB	238 MB	4	—
   C	6 m	259 MB	311 MB	3	4097

   A 6-minute-old process and a 3-hour-old process sit within 12% of each other — age is not the variable. ~4096 stacks x 32 KB is ~64 MB dirty plus ~33 MB of kernel page tables, on top of a ~150–190 MB interpreter/tree-sitter/driver baseline. That is the 250–350 MB plateau, and sampling every 2 min shows a sawtooth (stacks got reclaimed at one point, footprint fell 270 -> 192 MB, then climbed again) rather than a monotonic leak.

   Instead of bloating past the ceiling, the process dies: a burst wider than 4096 concurrent timers makes pthread_create fail and CPython raises RuntimeError: can't start new thread. Two watcher processes died within minutes of my probes, and macOS wrote no crash report because it is a clean Python exception exit.

   Hypothesis I could not test: Linux/WSL2 has no comparable low per-process thread cap (threads-max is typically in the tens of thousands and cgroup pids.max is often unlimited), and glibc's default thread stack is 8 MB. If the thread count there tracks the event rate with no 4096 ceiling to stop it, the same mechanism runs until RAM is gone rather than dying at a plateau — which matches "grows progressively until it exhausts all system memory (128 GB)" under Claude Code, an agent that generates continuous file churn. Someone on Linux could confirm by watching ls /proc/<pid>/task | wc -l during a git checkout of a large branch.

   Relationship to #1275 and #716

   #1275 fixed the correctness half of this — ignored paths no longer produce stale/duplicate graph nodes, because _handle_modification_locked checks _should_ignore. The cost half is still open: the thread is created, started, and scheduled before that check runs, so ignored paths are free in the graph but not free in memory.

   Suggested fixes

   1. Gate on_deleted like its siblings (small, removes the common trigger):

   def on_deleted(self, event):
       if event.is_directory:
           return
       p = Path(event.src_path)
       if p.suffix not in self.graph_builder.parsers or self._should_ignore(p):
           return
       self._debounce(event.src_path, lambda: self._handle_modification(event.src_path))

   Note it cannot reuse _is_supported_code_file verbatim: that calls path_obj.is_file(), which is always False for a path that was just deleted. I suspect that is why the filter was omitted.

   2. Stop using one thread per event (the actual fix). A single long-lived worker thread draining a {path: deadline} map costs one thread total regardless of event volume, and gives correct coalescing for free. As it stands, an N-file burst inside the debounce window costs N threads even when they all collapse to one unit of work — and past 4096 on macOS it is fatal rather than merely wasteful.

   Environment

   • codegraphcontext 0.5.7 (pipx), Python 3.14.6
   • macOS 15 (Darwin 25.5.0), Apple Silicon
   • Neo4j 2026.05.0, bolt://localhost:7687
   • Watched repo: polyglot Java / Dart / TypeScript monorepo, ~77k files, ~353k nodes indexed
   • cgc watch started on an already-indexed repo (perform_initial_scan=False, no --sync-on-start)

   One measurement note for anyone reproducing on macOS: ps RSS is misleading here — it reported 29 MB while /usr/bin/footprint -p <pid> reported 290 MB for the same process, because the thread stacks are compressed. Use footprint, and count regions with vmmap <pid> | grep -c '^Stack '. Also pipe ps -M into wc -l rather than reading it directly, or terminal truncation will show you 3 threads when there are 804.
10. Shashankss1205

    assigned

    Gurudutta22

    on Aug 13, 2026
11. Shashankss1205

    added

    gssoc:approvedGSSoC validation: counts toward scoringGSSoC validation: counts toward scoring

    level:criticalGSSoC difficulty: 80 pts contributor / 50 mentorGSSoC difficulty: 80 pts contributor / 50 mentor

    mentor:Shashankss1205GSSoC mentor attribution: credits reviewing mentorGSSoC mentor attribution: credits reviewing mentor

    type:bugGSSoC type bonus: bugGSSoC type bonus: bug

    on Sep 24, 2026
12. Shashankss1205 commented on Oct 15, 2026

    

    Shashankss1205

    on Oct 15, 2026 – with ChatGPT Codex Connector

    Collaborator

    More actions

    Maintenance review — 15 October 2026

    The reporter supplied a deterministic memory reproducer; use that to measure retained watcher data over repeated edits and deletions. Parsing parallelism in draft #1052 is not a memory-leak fix.