[codex] Parallelize incremental parse step - #1052

@Extrodox is attempting to deploy a commit to the shashankss1205's projects Team on Vercel.

A member of the Team first needs to authorize it.

Overlaps merged #1071 watcher sync and conflicts with current watcher.py.

Rebased onto current main and pushed to your branch — it was 64 days stale and watcher.py has had 15 commits since you branched from v0.4.13. Suite is green (1111 passed, 7 skipped). Two things I had to resolve:

1. The conflict in _handle_modification. Kept your side — capturing the return of update_file_in_graph to reuse the parse. I checked that still works: graph_builder.py:1015/1019 returns file_data, so the reuse optimisation is valid against current main.
2. Your new test failed with AttributeError: 'RepositoryEventHandler' object has no attribute '_update_lock'. main added _update_lock/_timers_lock in fix(watcher): handle both endpoints of a rename; serialise concurrent updates #1424 after you branched, and tests building the handler via __new__ must supply them — the neighbouring tests in that file already do. Added the same two lines.

Why I have not merged it

The parallel parsing hits an open segfault bug — #1370.

That issue reports non-deterministic Fatal Python error: Segmentation fault when multiple worker threads call tree_sitter_manager.execute_query() concurrently on different languages. Its traceback shows three threads inside execute_query() at once (TypeScript, Python, HTML).

Your ThreadPoolExecutor runs graph_builder.parse_file across affected files, and in a mixed-language repo the affected set spans languages — the exact condition. And execute_query is still unsynchronised on main:

# utils/tree_sitter_manager.py:322
def execute_query(language, query_string, node):
    query = Query(language, query_string)      # per call, no lock
    cursor = QueryCursor(query)
    res = cursor.captures(node)

tree_sitter_manager locks its language cache (_cache_lock) but nothing guards query execution.

So merging this would move a known crash from cgc index into cgc watch, where it is worse: watch runs unattended for hours, and a segfault takes the whole process down mid-session.

What would unblock it

Either is fine, and either would be a genuinely valuable PR on its own:

• Fix Bug: Segfault during 'cgc index' — concurrent tree-sitter execute_query() calls across worker threads (race condition in native extension) #1370 first — a module-level lock around query execution in execute_query, or a per-language lock if the contention matters. Then this merges as-is.
• Or bound the parallelism to be safe — e.g. group paths_to_parse by suffix and parse each language group serially while running groups concurrently, so no two threads are ever in the same language's grammar. That sidesteps Bug: Segfault during 'cgc index' — concurrent tree-sitter execute_query() calls across worker threads (race condition in native extension) #1370 without waiting on it, though it is a guess at the race's shape.

The rest of the PR is good — reusing the already-parsed changed file is a clear win regardless, and if you wanted to split that out as its own PR it would merge immediately. Keeping this open as a draft.

#1370 is now fixed — #1593 merged, adding a process-wide lock around tree-sitter query construction and execution.

That was the blocker on this PR. Measured cost of the lock is ~1% (3.48s vs 3.45s median over 120 parses on 8 threads), because only the query step is serialised — parser.parse and all Python-level work stay parallel, so your ThreadPoolExecutor still buys what it was meant to buy.

One caveat I want to be straight about: I could not reproduce the segfault locally, so #1593 is a guard against the reported crash rather than one I demonstrated. It is not proof your parallelism is now safe — it removes the specific unsynchronised path the report pointed at.

Your branch is already rebased onto main and green. Since it is still marked draft, I have left it for you to mark ready. Worth re-running your own workload against it first if you have the mixed-language repo that motivated the change.

This is still marked as a draft, so I've left it alone — flagging that it's now conflict-free against main if you'd like to mark it ready for review. Parallelising the incremental parse step is worth having; happy to review once you take it out of draft.

Heads-up: this is still marked as a draft, so I can't merge it — GitHub blocks the merge regardless of approval.

I've approved it and verified it independently (details in the review above): it applies cleanly to current main, and the full unit + integration suite is green — 1598 passed, 8 skipped, which is main's 1596 plus your two new tests, with no regressions.

Mark it ready for review and I'll merge it immediately. If the draft status was deliberate and there's something you still wanted to change, tell me what and I'll wait — but from here it looks finished.

@Extrodox — flagging you directly since this has been sitting for four months and a draft PR doesn't surface in the normal review queue. That's partly on me for not spotting it sooner.