cuda.core tests: memory_ipc/test_errors.py is not thread-safe under pytest-run-parallel (SEGV in cuMemPoolDestroy, shared harness state)

@rluo8

Summary

Two tests added by #2728 in cuda_core/tests/memory_ipc/test_errors.py fail on the free-threaded CI jobs (Test linux-aarch64 / Python 3.14t), where pytest-run-parallel executes each test body concurrently in several threads (the module sets pytestmark = pytest.mark.parallel_threads_limit(4)). One failure is a segmentation fault inside cuMemPoolDestroy; the other is a data race on the test class instance. Both reproduce on main itself, independent of the PR they were first noticed on.

Failure 1: SEGV in `cuMemPoolDestroy` from `test_allocation_handle_forking_pickler_roundtrip`

Observed on main for the #2728 merge commit bcfbe72c8c7 (run 34230002523, job 102076702071, CUDA 12.9.1 local, L4) and again on #2759 (run 34253228097, job 102163403217, CUDA 13.0.2 wheels, A100). The faulthandler dump is the same in both:

Fatal Python error: Segmentation fault
Stack (most recent call first):
  File ".../cuda_core/tests/memory_ipc/test_errors.py", line 327 in test_allocation_handle_forking_pickler_roundtrip
  File ".../cuda_core/tests/conftest.py", line 104 in wrapper
  File ".../pytest_run_parallel/plugin.py", line 83 in closure
Current thread's C stack trace (most recent call first):
  libcuda.so.1 ...
  libcuda.so.1, at cuMemPoolDestroy+0x30
  cuda/bindings/_bindings/cydriver...
  cuda/core/cu13/_resource_handles...
  cuda/core/cu13/_memory/_memory_pool...

Line 327 is the mr.close() in the test's finally. Each of the four threads creates its own IPC-enabled DeviceMemoryResource, reads mr.allocation_handle, round-trips it through multiprocessing.reduction.ForkingPickler, closes the restored handle, and destroys its pool. The crash is inside the driver's cuMemPoolDestroy while the other threads are exporting or destroying their own IPC pools. Whether this is a driver-level restriction on concurrent IPC pool export/destroy or a cuda.core teardown problem is not established; either way the test cannot run in parallel threads as written.

Failure 2: `TestImportWrongMR.test_main[PinnedMR]` pickles another thread's closed resource

Observed on #2759 (run 34253228097, job 102163403408, CUDA 13.3.0 local, L4). The test is marked flaky(reruns=2) and failed all three attempts:

RuntimeError: DeviceMemoryResource has been closed
  when serializing cuda.core._memory._buffer.Buffer object
  when serializing dict item 'buffer'
  when serializing test_errors.TestImportWrongMR state
  when serializing multiprocessing.context.Process object

ChildErrorHarness.test_main stores its fixtures on the instance (self.device, self.mr, self._extra_mrs, and self.buffer set by PARENT_ACTION) and spawns the child with target=self.child_main, which pickles self. pytest-run-parallel runs the four threads against the same test instance and the same function-scoped ipc_memory_resource value, so one thread pickles a Buffer whose resource another thread has already closed. The same shared-state pattern applies to every subclass of ChildErrorHarness in the file; this one just lost the race first.

Suggested fix

Mark the ChildErrorHarness subclasses and test_allocation_handle_forking_pickler_roundtrip @pytest.mark.thread_unsafe (reason: shared instance state and multiprocessing spawn; concurrent IPC pool export/destroy), or drop the module-level parallel_threads_limit(4) in favour of per-test marks.
Separately investigate whether concurrent cuMemPoolDestroy of IPC-enabled pools from several threads is supposed to be safe; if it is, the SEGV is a cuda.core bug in pool teardown rather than a test-only issue.

Environment

GitHub Actions, linux-aarch64, Python 3.14.7 free-threaded, cuda-bindings 13.x, pytest-run-parallel, CUDA 12.9.1 / 13.0.2 / 13.3.0.

Refs: #2728 (tests added), #2759 (where the failures were analysed).

@rluo8 and @seberg for viz

@Andy-Jost

Hi @Andy-Jost @seberg , I opened a PR #2785 to serialize the thread-unsafe tests in test_errors.py. It does not fix the cuMemPoolDestroy SEGV root cause. Thanks!

____________ ERROR at call of TestImportWrongMR.test_main[PinnedMR] ____________

self = <test_errors.TestImportWrongMR object at 0x475f143b190>
ipc_device = <Device 0 (NVIDIA L4)>
ipc_memory_resource = <cuda.core._memory._pinned_memory_resource.PinnedMemoryResource object at 0x475f4d033c0>

    @pytest.mark.flaky(reruns=2)
    def test_main(self, ipc_device, ipc_memory_resource):
        """Parent process that checks child errors."""
        # Attach fixtures to this object for convenience. These can be accessed
        # from PARENT_ACTION.
        self.device = ipc_device
        self.mr = ipc_memory_resource
        self._extra_mrs = []
    
        try:
            # Start a child process to generate error info.
            pipe = [multiprocessing.Queue() for _ in range(2)]
            process = multiprocessing.Process(target=self.child_main, args=(pipe, self.device, self.mr))
>           process.start()

ipc_device = <Device 0 (NVIDIA L4)>
ipc_memory_resource = <cuda.core._memory._pinned_memory_resource.PinnedMemoryResource object at 0x475f4d033c0>
pipe       = [<multiprocessing.queues.Queue object at 0x475f60c0310>, <multiprocessing.queues.Queue object at 0x475f60c0410>]
process    = <Process name='Process-29' parent=16658 initial>
self       = <test_errors.TestImportWrongMR object at 0x475f143b190>

tests/memory_ipc/test_errors.py:134: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.15.0-rc.2/x64-freethreaded/lib/python3.15t/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
        self       = <Process name='Process-29' parent=16658 initial>
/opt/hostedtoolcache/Python/3.15.0-rc.2/x64-freethreaded/lib/python3.15t/multiprocessing/context.py:233: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        process_obj = <Process name='Process-29' parent=16658 initial>
/opt/hostedtoolcache/Python/3.15.0-rc.2/x64-freethreaded/lib/python3.15t/multiprocessing/context.py:297: in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
        Popen      = <class 'multiprocessing.popen_spawn_posix.Popen'>
        process_obj = <Process name='Process-29' parent=16658 initial>
/opt/hostedtoolcache/Python/3.15.0-rc.2/x64-freethreaded/lib/python3.15t/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
        __class__  = <class 'multiprocessing.popen_spawn_posix.Popen'>
        process_obj = <Process name='Process-29' parent=16658 initial>
        self       = <multiprocessing.popen_spawn_posix.Popen object at 0x475f6200710>
/opt/hostedtoolcache/Python/3.15.0-rc.2/x64-freethreaded/lib/python3.15t/multiprocessing/popen_fork.py:20: in __init__
    self._launch(process_obj)
        process_obj = <Process name='Process-29' parent=16658 initial>
        self       = <multiprocessing.popen_spawn_posix.Popen object at 0x475f6200710>
/opt/hostedtoolcache/Python/3.15.0-rc.2/x64-freethreaded/lib/python3.15t/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
        fp         = <_io.BytesIO object at 0x475f62415c0>
        prep_data  = {'log_to_stderr': False, 'authkey': b'\xbf&o6\xe3k\xa3\x88\xdb4\xdf\xb2k\x9c\xe2\xeb;\xb80\xdbH\x83}p\x7f\\\x12\xa5a\x...uda-python/cuda-python/cuda_core/tests/graph', '/__w/cuda-python/cuda-python/cuda_core/tests/example_tests', ...], ...}
        process_obj = <Process name='Process-29' parent=16658 initial>
        resource_tracker = <module 'multiprocessing.resource_tracker' from '/opt/hostedtoolcache/Python/3.15.0-rc.2/x64-freethreaded/lib/python3.15t/multiprocessing/resource_tracker.py'>
        self       = <multiprocessing.popen_spawn_posix.Popen object at 0x475f6200710>
        tracker_fd = 57
/opt/hostedtoolcache/Python/3.15.0-rc.2/x64-freethreaded/lib/python3.15t/multiprocessing/reduction.py:60: in dump
    ForkingPickler(file, protocol).dump(obj)
        file       = <_io.BytesIO object at 0x475f62415c0>
        obj        = <Process name='Process-29' parent=16658 initial>
        protocol   = None
cuda/core/_memory/_buffer.pyx:342: in cuda.core._memory._buffer.Buffer.__reduce__
    return Buffer._reduce_helper, (self.memory_resource, self.ipc_descriptor)
cuda/core/_memory/_buffer.pyx:417: in cuda.core._memory._buffer.Buffer.ipc_descriptor.__get__
    ipc_data = IPCDataForBuffer(_ipc.Buffer_get_ipc_descriptor(self), False)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise RuntimeError("Memory resource is not IPC-enabled")
E   RuntimeError: Memory resource is not IPC-enabled
E   when serializing dict item 'buffer'
E   when serializing test_errors.TestImportWrongMR state
E   when serializing test_errors.TestImportWrongMR object
E   when serializing tuple item 0
E   when serializing method reconstructor arguments
E   when serializing method object
E   when serializing dict item '_target'
E   when serializing multiprocessing.context.Process state
E   when serializing multiprocessing.context.Process object


cuda/core/_memory/_ipc.pyx:177: RuntimeError

Tests serialized in #2785 (merged). ChildErrorHarness.test_main, test_from_allocation_handle_raw_fd_imports_mapped_pool, test_allocation_handle_forking_pickler_roundtrip, and test_ipc_registry_dedups_repeated_imports are now marked thread_unsafe so CI is unblocked. Root cause (SEGV in cuMemPoolDestroy under concurrent IPC pool export/destroy, and the shared-harness-state pickling issue) is still open; this issue stays open to track that. (#2791 closed as a duplicate/superseded fix.)

Closing. Root-caused to the CUDA driver, not cuda.core: the crash reproduces with a driver-API-only C program and no Python involved. #2785 keeps the affected tests serialized. Follow-up is happening outside this tracker.

SEGV tracked internally as nvbug 6763003

Navigation Menu

Repository navigation

cuda.core tests: memory_ipc/test_errors.py is not thread-safe under pytest-run-parallel (SEGV in cuMemPoolDestroy, shared harness state) #2784

Summary

Failure 1: SEGV in `cuMemPoolDestroy` from `test_allocation_handle_forking_pickler_roundtrip`

Failure 2: `TestImportWrongMR.test_main[PinnedMR]` pickles another thread's closed resource

Suggested fix

Environment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

cuda.core tests: memory_ipc/test_errors.py is not thread-safe under pytest-run-parallel (SEGV in cuMemPoolDestroy, shared harness state) #2784

Description

Summary

Failure 1: SEGV in cuMemPoolDestroy from test_allocation_handle_forking_pickler_roundtrip

Failure 2: TestImportWrongMR.test_main[PinnedMR] pickles another thread's closed resource

Suggested fix

Environment

Activity

rluo8 commented on Sep 9, 2026

leofang commented on Sep 9, 2026

Andy-Jost commented on Sep 9, 2026

Andy-Jost commented on Sep 10, 2026

Andy-Jost commented on Sep 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Failure 1: SEGV in `cuMemPoolDestroy` from `test_allocation_handle_forking_pickler_roundtrip`

Failure 2: `TestImportWrongMR.test_main[PinnedMR]` pickles another thread's closed resource