CI: Random segfaults on Windows #3038

Activity

1. leofang

   added

   bugSomething isn't workingSomething isn't working

   P0High priority - Must do!High priority - Must do!

   cuda.coreEverything related to the cuda.core moduleEverything related to the cuda.core module

   on Oct 6, 2026
2. leofang commented on Oct 6, 2026

   

   leofang

   on Oct 6, 2026

   MemberAuthor

   More actions

   Scanning the CI workflow runs between 2026-09-29 and 2026-10-06, I found 5 more Windows access violation (STATUS_ACCESS_VIOLATION, 0xC0000005) instances matching this signature, every one on free-threaded Python 3.14t + a100 MCDM:

   Date UTC	Context	SHA	CUDA	Crashing test	Job
   2026-10-02 16:57	#2947	e2ae412	13.4.2 wheels	test_linker.py::test_incremental_ltoir_round_trip	log
   2026-10-03 03:47	#3013	b8b2ffe	13.0.2 wheels	test_linker.py::test_incremental_ltoir_round_trip	log
   2026-10-05 00:38	scheduled on main	ab9e20f	13.0.2 wheels	test_memory.py::test_device_memory_resource_with_options	log
   2026-10-06 17:24	#3028	ef9b712	13.4.2 wheels	test_linker.py::test_incremental_ltoir_round_trip	log
   2026-10-06 22:14	#3037 (this issue)	8deb152	13.4.2 wheels	test_linker.py::test_incremental_ltoir_round_trip	log

   Observations:

   • Every hit is win-64 + Python 3.14t + a100 MCDM. Zero access violation on 3.10–3.14 (non-free-threaded) or 3.15 across all 48 Windows Run cuda.core tests failures in the window — free-threading looks necessary to trigger.
   • 4/5 crash in test_incremental_ltoir_round_trip — strongest reproducer candidate.

   For reference, in the same window two additional Windows failures surfaced Windows fatal exception: code 0xC0000374 (STATUS_HEAP_CORRUPTION, exit 127) rather than access violation + 139 — different code, but same environment (win-64 + 3.14t + a100 MCDM), so a shared memory-safety root cause is plausible:

   • 2026-09-30 — build: consolidate toolchain helpers via symlink #2965 → log, crash in test_linker.py::test_incremental_lto_cubin_round_trip
   • 2026-10-02 — ci: keep test failures visible in GitHub Actions logs #2963 → log, crash in graph/test_graph_definition_lifetime.py::test_destroying_conditional_node_invalidates_branch_handles

   -- Leo's bot
3. leofang commented on Oct 6, 2026

   

   leofang

   on Oct 6, 2026

   Last edited by leofang

   MemberAuthor

   More actions

   Correction to my earlier comment: the sweep filtered by step-number instead of step-name and missed 2 instances where "Run cuda.core tests" was numbered 31 (not 32). Both newly surfaced hits are Python 3.13 (non-free-threaded) + rtxpro6000 MCDM, so free-threading is not required to trigger — contrary to what I wrote above.

   Date UTC	Py	Context	SHA	CUDA / GPU	Crashing test	Job
   2026-09-29 20:33	3.13	scheduled on main	e8d9075	13.0.2 wheels / rtxpro6000 MCDM	test_green_context.py::test_memory_node_updates_preserve_green_context (crash in the green_ctx fixture)	log
   2026-10-01 19:56	3.13	#2988	144e356	13.0.2 wheels / rtxpro6000 MCDM	test_green_context.py::test_memory_node_updates_preserve_green_context (same green_ctx fixture)	log

   -- Leo's bot
4. rparolin

   added this to the cuda.core 1.3.0 milestone on Oct 8, 2026
5. rparolin

   assigned

   Andy-Jost

   on Oct 8, 2026
6. rparolin

   added the

   Bug

   issue type on Oct 8, 2026