Implement a handle-based resource management layer

Implement a new handle-based resource management layer in the cuda.core Python package, replacing direct lifetime management via __del__, __dealloc__, and similar destructors with the handle abstraction described in the attached proposal. The task includes updating existing CUDA resource wrappers (e.g., contexts, streams, device pointers) to own and interact with opaque handles rather than raw CUDA resources, ensuring referentially transparent lifetimes, structural dependency tracking, and correct automatic release when handles become unreachable.

Proposal_ Handle Abstraction for CUDA Resource Management in cuda-core.pdf

@Andy-Jost

I've seen noisy tracebacks like those below many times when testing interactively.

@Andy-Jost Do you want to track working on those here, or under a separate issue?

tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main Traceback (most recent call last):
  File "cuda/core/experimental/_memory/_device_memory_resource.pyx", line 593, in cuda.core.experimental._memory._device_memory_resource.DMR_close
    HANDLE_RETURN(cydriver.cuMemPoolDestroy(self._handle))
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 63, in cuda.core.experimental._utils.cuda_utils.HANDLE_RETURN
    return _check_driver_error(err)
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 81, in cuda.core.experimental._utils.cuda_utils._check_driver_error
    raise CUDAError(f"{name.decode()}: {expl}")
cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
Exception ignored in: 'cuda.core.experimental._memory._device_memory_resource.DeviceMemoryResource.__dealloc__'
Traceback (most recent call last):
  File "cuda/core/experimental/_memory/_device_memory_resource.pyx", line 593, in cuda.core.experimental._memory._device_memory_resource.DMR_close
    HANDLE_RETURN(cydriver.cuMemPoolDestroy(self._handle))
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 63, in cuda.core.experimental._utils.cuda_utils.HANDLE_RETURN
    return _check_driver_error(err)
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 81, in cuda.core.experimental._utils.cuda_utils._check_driver_error
    raise CUDAError(f"{name.decode()}: {expl}")
cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
PASSED
tests/memory_ipc/test_serialize.py::TestObjectSerializationWithMR::test_main Error in sys.excepthook:

Original exception was:
PASSED
tests/memory_ipc/test_serialize.py::TestObjectPassing::test_main Traceback (most recent call last):
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 63, in cuda.core.experimental._utils.cuda_utils.HANDLE_RETURN
    return _check_driver_error(err)
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 81, in cuda.core.experimental._utils.cuda_utils._check_driver_error
    raise CUDAError(f"{name.decode()}: {expl}")
cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
Exception ignored in: 'cuda.core.experimental._memory._device_memory_resource.DMR_deallocate'
Traceback (most recent call last):
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 63, in cuda.core.experimental._utils.cuda_utils.HANDLE_RETURN
    return _check_driver_error(err)
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 81, in cuda.core.experimental._utils.cuda_utils._check_driver_error
    raise CUDAError(f"{name.decode()}: {expl}")
cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
Traceback (most recent call last):
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 63, in cuda.core.experimental._utils.cuda_utils.HANDLE_RETURN
    return _check_driver_error(err)
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 81, in cuda.core.experimental._utils.cuda_utils._check_driver_error
    raise CUDAError(f"{name.decode()}: {expl}")
cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
Exception ignored in: 'cuda.core.experimental._memory._device_memory_resource.DMR_deallocate'
Traceback (most recent call last):
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 63, in cuda.core.experimental._utils.cuda_utils.HANDLE_RETURN
    return _check_driver_error(err)
  File "cuda/core/experimental/_utils/cuda_utils.pyx", line 81, in cuda.core.experimental._utils.cuda_utils._check_driver_error
    raise CUDAError(f"{name.decode()}: {expl}")
cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
Error in sys.excepthook:

Original exception was:
PASSED

FWIW — Interestingly, the __dealloc__ tracebacks do not show up in our CI logs. I looked a couple weeks ago, and now again.

I really wonder what's different in the interactive tests.

Details:

Using the log archive from this run:

https://github.com/NVIDIA/cuda-python/actions/runs/19943974705

rwgk-win11.localdomain:~/logs_19943974705 $ grep -a __dealloc__ *.txt

(no output)

rwgk-win11.localdomain:~/logs_19943974705 $ grep -a 'most recent call last' *.txt | cut -d: -f1 | uniq -c
     50 Docs___Build_docs.txt

rwgk-win11.localdomain:~/logs_19943974705 $ grep -a tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main *.txt
Test_linux-64___py3.10__12.9.1__local__v100.txt:2025-12-04T21:21:27.1591125Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.10__13.0.2__wheels__l4.txt:2025-12-04T21:23:41.4318068Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.10__13.1.0__wheels__l4.txt:2025-12-04T21:26:52.8617594Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.11__12.9.1__wheels__rtxpro6000.txt:2025-12-04T21:27:15.8292581Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.11__13.0.2__local__l4.txt:2025-12-04T21:24:53.9694002Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.11__13.1.0__local__l4.txt:2025-12-04T21:24:58.9242299Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.12__12.9.1__local__l4.txt:2025-12-04T21:29:56.1703542Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.12__13.0.2__wheels__l4.txt:2025-12-04T21:24:17.0893008Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.12__13.1.0__wheels__l4.txt:2025-12-04T21:24:18.1975803Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.13__12.9.1__wheels__v100.txt:2025-12-04T21:21:23.2275674Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.13__13.0.2__local__H100.txt:2025-12-04T21:22:03.6042416Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.13__13.0.2__local__rtxpro6000.txt:2025-12-04T21:32:40.0504863Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.13__13.1.0__local__H100.txt:2025-12-04T21:22:25.9978795Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.13__13.1.0__local__rtxpro6000.txt:2025-12-04T21:34:54.4827695Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.14__13.0.2__local__l4.txt:2025-12-04T21:28:41.2306884Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.14__13.1.0__local__l4.txt:2025-12-04T21:24:22.3124956Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.14t__13.0.2__local__l4.txt:2025-12-04T21:25:28.3826057Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-64___py3.14t__13.1.0__local__l4.txt:2025-12-04T21:25:13.4215794Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.10__12.9.1__local__a100.txt:2025-12-04T21:37:23.1336724Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.10__13.0.2__wheels__a100.txt:2025-12-04T21:38:13.5627346Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.10__13.1.0__wheels__a100.txt:2025-12-04T21:44:55.1105628Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.11__12.9.1__wheels__a100.txt:2025-12-04T21:44:10.1873848Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.11__13.0.2__local__a100.txt:2025-12-04T21:44:22.7745884Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.11__13.1.0__local__a100.txt:2025-12-04T21:38:52.6202987Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.12__12.9.1__local__a100.txt:2025-12-04T21:51:48.7308322Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.12__13.0.2__wheels__a100.txt:2025-12-04T21:42:04.2526405Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.12__13.1.0__wheels__a100.txt:2025-12-04T21:42:47.5277575Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.13__12.9.1__wheels__a100.txt:2025-12-04T21:45:37.5213893Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.13__13.0.2__local__a100.txt:2025-12-04T21:46:36.4146672Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.13__13.1.0__local__a100.txt:2025-12-04T21:44:10.2815064Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.14__13.0.2__local__a100.txt:2025-12-04T21:47:59.1542422Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.14__13.1.0__local__a100.txt:2025-12-04T21:47:25.1348576Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.14t__13.0.2__local__a100.txt:2025-12-04T21:37:00.2298688Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_linux-aarch64___py3.14t__13.1.0__local__a100.txt:2025-12-04T21:31:57.7789946Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main PASSED [  8%]
Test_win-64___py3.10__12.9.1__wheels__rtx2080__WDDM_.txt:2025-12-04T21:34:55.1947681Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.10__13.0.2__local__rtxpro6000__TCC_.txt:2025-12-04T21:48:36.9307217Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.10__13.1.0__local__rtxpro6000__TCC_.txt:2025-12-04T21:44:53.1781296Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.11__12.9.1__local__v100__MCDM_.txt:2025-12-04T21:32:18.1553290Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.11__13.0.2__wheels__rtx4090__WDDM_.txt:2025-12-04T21:30:05.5174678Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.11__13.1.0__wheels__rtx4090__WDDM_.txt:2025-12-04T21:29:44.2063998Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.12__12.9.1__wheels__l4__MCDM_.txt:2025-12-04T21:32:36.3838862Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.12__13.0.2__local__a100__TCC_.txt:2025-12-04T21:32:48.0137350Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.12__13.1.0__local__a100__TCC_.txt:2025-12-04T21:32:39.6758111Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.13__12.9.1__local__l4__TCC_.txt:2025-12-04T21:33:58.5118452Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.13__13.0.2__wheels__rtxpro6000__MCDM_.txt:2025-12-04T21:57:35.2189670Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.13__13.1.0__wheels__rtxpro6000__MCDM_.txt:2025-12-04T21:36:47.7639407Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.14__12.9.1__wheels__v100__TCC_.txt:2025-12-04T21:39:44.5438503Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.14__13.0.2__local__l4__MCDM_.txt:2025-12-04T21:34:04.7308079Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.14__13.1.0__local__l4__MCDM_.txt:2025-12-04T21:37:42.4736675Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.14t__12.9.1__local__l4__TCC_.txt:2025-12-04T21:37:20.9570565Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.14t__13.0.2__wheels__a100__MCDM_.txt:2025-12-04T21:27:24.2010849Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]
Test_win-64___py3.14t__13.1.0__wheels__a100__MCDM_.txt:2025-12-04T21:32:38.0499609Z tests/memory_ipc/test_serialize.py::TestObjectSerializationDirect::test_main SKIPPED [  8%]

@rwgk

Hi @rwgk, yes it would be helpful to track these noisy tracebacks here. If we have instances that can be reliably reproduced, then they will help. This work is intended to fix those problems.

I really wonder what's different in the interactive tests

In some cases when running under pytest, the -s switch is needed for __dealloc__ problems to surface.

I really wonder what's different in the interactive tests

In some cases when running under pytest, the -s switch is needed for __dealloc__ problems to surface.

Oh, that's probably it!

In the QA testing I always use pytest -ra -s -v

Navigation Menu

Repository navigation

Implement a handle-based resource management layer #1303

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Implement a handle-based resource management layer #1303

Description

Activity

rwgk commented on Dec 5, 2025

rwgk commented on Dec 5, 2025

Andy-Jost commented on Dec 5, 2025

Andy-Jost commented on Dec 5, 2025

rwgk commented on Dec 5, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions