[BUG]: Intermittent CUDA_ERROR_OUT_OF_MEMORY cascade in cuda_core test suite #2381

Activity

1. rluo8

   added

   bugSomething isn't workingSomething isn't working

   on Jul 17, 2026
2. rparolin

   assigned

   juenglin

   on Jul 23, 2026
3. rparolin

   added the

   Bug

   issue type on Jul 23, 2026
4. rparolin

   added this to the cuda.core 1.2.0 milestone on Jul 23, 2026
5. Andy-Jost commented on Aug 3, 2026

   

   Andy-Jost

   on Aug 3, 2026

   Contributor

   More actions

   Root cause

   Each CUDA memory pool reserves virtual address space at creation. Reservations are address space only — no physical memory is committed — but the per-process VA budget is bounded, and on Windows (notably MCDM) that bound is roughly 1 TB rather than the 47+ bits available on Linux.

   Three factors combine:

   1. Pools created without max_size get a system-dependent reservation. That default scales with device memory and lands in the hundreds of gigabytes on large-memory GPUs. Most cuda.core tests pass max_size=2097152 and are harmless; the ones that pass options without setting it (the IPC tests, primarily) reserve a full-size window for a pool that holds kilobytes. The cuda.bindings mempool tests construct pool properties by hand and leave maxSize at zero, with the same effect.

   2. Pool destruction is not synchronous with respect to address-space release. Teardown blocks on the stream-ordered frees of the pool's outstanding allocations, which are themselves queued behind pending GPU work. Dropping the Python object marks the pool dead but does not return its reservation.

   3. Test boundaries don't force that drain. Fixtures release memory resources and move on, so reservations from prior tests remain live while the next test creates its own. Over a session these accumulate until a creation — or a lazy default-pool lookup — fails with CUDA_ERROR_OUT_OF_MEMORY on a device with essentially all of its memory free.

   Observed symptoms follow directly

   Randomized ordering (pytest-randomly) determines which test is holding reservations when the budget is exhausted, so the failure migrates between runs. Isolated reruns pass because the process starts clean. A 2–3 second inter-test sleep mostly fixes it because it lets queued work and pending frees retire, at which point the abandoned pools actually tear down. Failures surface in whichever test asks next, which is why they cluster in graph and mempool tests that allocate only a few bytes — the allocation size is never the issue.

   Scope

   This accounts for the failures reported here. A few of them are tests asserting a specific error (e.g. "not supported") that instead receive the OOM, so they are masked rather than caused by their own logic. Note also that retry logic on the default device-pool lookup alone is insufficient: explicit pool creation, graph-pool initialization, and pinned/managed lookup fail identically.

   Suggested fixes

   1. Audit every pool construction in both test suites for a missing max_size and set an explicit small cap wherever the test doesn't specifically need the default.
   2. Make the memory-resource fixtures release deterministically — synchronize before dropping the resource — so reservations don't leak across test boundaries.
6. juenglin commented on Aug 6, 2026

   

   juenglin

   on Aug 6, 2026

   Contributor

   More actions

   Revised root cause thesis: two driver-managed memory pools, each permanently reserving 2x device memory of virtual address space

   Measured by walking the process address space with Windows VirtualQuery (exact, non-invasive)
   and driving the driver API directly, so cuda.core is not in the path. On a 23.89 GiB device:

   Driver call	VA reserved	Reclaimable?
   cuDeviceGetMemPool	+47.78 GiB (2.00x device mem)	never (process lifetime)
   cuMemAllocFromPoolAsync (first alloc)	0	—
   cuGraphAddMemAllocNode	+47.78 GiB (2.00x device mem)	never — not by cuGraphDestroy, cuGraphExecDestroy, or cuDeviceGraphMemTrim
   cuMemPoolCreate(maxSize=0)	+47.78 GiB (2.00x)	yes, on cuMemPoolDestroy
   cuMemPoolCreate(maxSize=2 MiB)	+1.00 GiB	yes, on cuMemPoolDestroy

   Key points:

   1. Merely querying the default pool costs 2x device memory. cuDeviceGetMemPool makes the
      reservation; the subsequent allocation is free. This is exactly the call that fails in the
      captured diagnostics (cuDeviceGetMemPool(dev 0) -> CUDA_ERROR_OUT_OF_MEMORY) and behind the
      DeviceMemoryResource(dev) line where every failing run first dies.
   2. The graph memory pool is a second, distinct driver-managed pool, reserved at node creation
      time and never released. It is per-device and shared: adding alloc nodes to a 2nd and 3rd graph
      reserved 0 additional bytes.
   3. Neither pool can be capped or substituted. cuGraphAddMemAllocNode takes no pool handle —
      only a CUmemPoolProps description. Setting poolProps.maxSize = 2 MiB is accepted and reads
      back correctly via cuGraphMemAllocNodeGetParams, but the reservation is still 2.00x device
      memory. The maxSize cap that works for cuMemPoolCreate is ignored here.

   Documentation references

   CUDA Programming Guide 4.2.5.5, Physical Memory Footprint:

   Note that cudaDeviceGraphMemTrim operates on a different pool from cudaMemPoolTrimTo().
   The graph memory pool is not exposed to the stream ordered memory allocator. [...]
   cudaDeviceGraphMemTrim will unmap and release any physical memory reserved by graph memory
   nodes that is not actively in use. [...] cudaGraphMemAttrReservedMemCurrent returns the amount
   of physical memory reserved by the driver for graph allocations.

   Same guide, on when the address is assigned:

   CUDA assigns virtual addresses for the graph allocation at node creation time.
   Driver API, cuDeviceGraphMemTrim:
   Blocks which are not in use by a graph that is either currently executing or scheduled to execute
   are freed back to the operating system.
   Driver API, cuMemAllocFromPoolAsync
   — stream capture is not a back door either:
   During stream capture, this function results in the creation of an allocation node. In this case,
   the allocation is owned by the graph instead of the memory pool. The memory pool's properties
   are used to set the node's creation parameters.

   Why this explains the intermittent cascade

   On the failing machine (178.81 GiB device), each reservation is 357.6 GiB; the two together are
   715 GiB, on top of the address space consumed by context initialization. Against the bounded
   per-process VA budget on Windows MCDM this leaves essentially no margin, so whether the second
   reservation finds a contiguous hole depends on test order and fragmentation — which is exactly the
   observed ~50% failure rate. It also explains the failure point: the cascade always begins at the
   first test needing the default pool after the graph pool has taken its 357.6 GiB.

   Note: cuda.core is not doing anything wrong here, and capping pools the tests create (#2484) was
   correct but addresses a smaller, separate cost.

   Possible workaround: materialize both pools at session start

   Total consumption is fixed and unavoidable, but when it happens is not. Both reservations are
   one-time-per-process, so a session-scoped autouse fixture that does one tiny allocation from the
   device default pool and creates one throwaway graph mem-alloc node would force both 2x windows to be
   reserved back-to-back into a nearly empty, unfragmented address space, before test order can
   interfere.

   This does not reduce the footprint; it makes the outcome deterministic. It is directly falsifiable:
   if the failures are caused by the second reservation failing to find a contiguous hole, this should
   convert an intermittent failure into a consistent pass or a consistent, immediate failure at session
   start, which would be equally informative.
7. juenglin

   added 3 commits that reference this issue on Aug 6, 2026

   test(cuda_core): reserve the driver's two pool address windows at ses…

   ...

   8567243

   test(cuda_core): reserve the driver's two pool address windows at ses…

   ...

   540d408

   test(cuda_core): reserve the driver's two pool address windows at ses…

   ...

   0e1c0fc
8. juenglin commented on Aug 11, 2026

   

   juenglin

   on Aug 11, 2026

   Contributor

   More actions

   Update: ASLR explains the intermittency

   On Windows, bottom-up ASLR randomizes where VirtualAlloc placements start
   on each launch. This carves the free address space into differently sized
   pieces. With a ~1 TB budget and two 357.6 GiB reservations to place, whether
   enough sufficiently wide address ranges exist depends on where the carving lands —
   which is why the same suite passes on one launch and fails on the next, uncorrelated
   with Python version or test order.

   Disabling bottom-up ASLR (Set-ProcessMitigation -Name python.exe -Disable BottomUp,HighEntropy) removes the carving and makes the suite deterministically
   pass: 12 of 12 sessions clean across two full runs, versus ~50% failure rate
   with ASLR on.
9. juenglin commented on Aug 12, 2026

   

   juenglin

   on Aug 12, 2026

   Contributor

   More actions

   @rluo8 confirmed that turning off ASLR on the Windows 2025 server prevents the intermittent failures with CUDA_ERROR_OUT_OF_MEMORY. Closing the issue.
10. juenglin

    closed this as completedon Aug 12, 2026
11. rluo8 commented on Aug 13, 2026

    

    rluo8

    on Aug 13, 2026

    ContributorAuthor

    More actions

    Thanks @juenglin .
12. juenglin

    added 5 commits that reference this issue on Oct 7, 2026

    test(cuda.core): freeze the collected heap once so the per-test gc.co…

    ...

    05a309f

    Remove xfail_if_mempool_oom and xfail_on_graph_mempool_oom helpers (#…

    ...

    197f773

    test(cuda.core): count live owned pools and VA reservations in C++

    ...

    0651f45

    test(cuda.core): gate the init_cuda teardown gc.collect() on live own…

    ...

    0271e3a

    test(cuda.core): pop the context when the leftover warning aborts ini…

    ...

    a32461f