Repository navigation
[BUG]: VirtualMemoryResource host_numa allocations always fail #2694
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct
Type of Bug
Runtime Error
Component
cuda.core
Describe the bug
Every VirtualMemoryResource allocation with location_type="host_numa" fails with CUDA_ERROR_INVALID_VALUE, on a system that reports host_numa_id 0 and host_numa_virtual_memory_management_supported True. VirtualMemoryResource.allocate passes prop.location.id = -1 whenever the resource has no device, which __init__ sets for every host location type, and cuMemCreate rejects CU_MEM_LOCATION_TYPE_HOST_NUMA with that id. cuda_core/cuda/core/typing.py documents HOST_NUMA as "host memory pinned to a specific NUMA node", and VirtualMemoryResourceOptions has no field that carries a node id.
location_type="host_numa_current" also fails, for a separate reason: cuMemCreate rejects CU_MEM_LOCATION_TYPE_HOST_NUMA_CURRENT for every node id I passed it, including 0.
test_vmm_host_location_types_report_host_accessible in cuda_core/tests/test_memory.py parametrizes over host, host_numa and host_numa_current, but it only checks mr.device and mr.is_host_accessible, so it stays green without ever calling allocate.
How to Reproduce
from cuda.core import Device, VirtualMemoryResource, VirtualMemoryResourceOptions
dev = Device()
dev.set_current()
print("host_numa_id =", dev.properties.host_numa_id)
for location_type in ("host", "host_numa"):
opts = VirtualMemoryResourceOptions(location_type=location_type, handle_type=None)
buf = VirtualMemoryResource(dev, config=opts).allocate(4096)
print(location_type, "allocated", buf.size, "bytes")
buf.close()host_numa_id = 0
host allocated 2097152 bytes
Traceback (most recent call last):
File "/tmp/vmm_host_numa.py", line 8, in <module>
buf = VirtualMemoryResource(dev, config=opts).allocate(4096)
File "/home/vyron-vasileiadis/projects/forks/cuda-python/cuda_core/cuda/core/_memory/_virtual_memory_resource.py", line 555, in allocate
raise_if_driver_error(res)
~~~~~~~~~~~~~~~~~~~~~^^^^^
File "cuda/core/_utils/cuda_utils.pyx", line 134, in cuda.core._utils.cuda_utils._check_driver_error
File "cuda/core/_utils/cuda_utils.pyx", line 145, in cuda.core._utils.cuda_utils._check_driver_error
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
Expected behavior
allocate should either return a buffer pinned to a NUMA node, the way location_type="host" and location_type="device" already do, or fail at construction with an error naming the missing node id. Driving cuMemCreate directly on this system, CU_MEM_LOCATION_TYPE_HOST_NUMA returns CUDA_SUCCESS at location.id = 0 and CUDA_ERROR_INVALID_VALUE at location.id = -1, so the location id is the only thing standing between the current behaviour and a working allocation.
Operating System
Ubuntu 26.04 LTS
nvidia-smi output
Tue Aug 25 11:22:23 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.84 Driver Version: 595.84 CUDA Version: 13.2 |
+-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3050 ... Off | 00000000:01:00.0 Off | N/A |
| N/A 46C P8 3W / 30W | 11MiB / 4096MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
Activity
somtri commentedon Sep 19, 2026
I'd like to investigate the host-NUMA allocation failure and add a test that calls allocate.
Andy-Jost commentedon Oct 5, 2026
@somtri Thanks, the issue is yours.
What goes wrong: the resource drops its device for every host location type, and
_fill_propthen writesprop.location.id = -1.cuMemCreaterequires the NUMA node id forCU_MEM_LOCATION_TYPE_HOST_NUMA(0on a system without NUMA) and rejectsCU_MEM_LOCATION_TYPE_HOST_NUMA_CURRENTfor physical allocations, so neither type can allocate today.The fix we expect:
VirtualMemoryResourceOptions, for examplenuma_id: int | None = None. Require it forlocation_type="host_numa", reject it for every other location type in_check_config, and write it toprop.location.idin_fill_prop. The access descriptors copy that location, and the driver ignores the id of aHOST_NUMAaccess entry, so_fill_accessneeds no change.location_type="host_numa_current"in_check_configwith aValueErrorthat says the driver does not accept it for physical allocations. Keep the enum member.cuda_core/tests/test_memory.py, next totest_vmm_host_location_types_report_host_accessible: one that allocates withhost_numaanddevice.properties.host_numa_id, copies data through the buffer, and closes it without aCUDAWarning(skip whenproperties.host_numa_virtual_memory_management_supportedis false orhost_numa_idis negative); one thathost_numa_currentraises at construction; one that a missing or misplacednuma_idraises. VMM tests call the real driver: no monkeypatching ofdriver.*and no hand-builtBufferobjects.cuda_core/docs/source/release/1.3.0-notes.rst.Keep the other host-location gaps (
device_idignored for host resources,peerson a host resource) out of this change; they are item 7 of #2910.