Skip to content

test_vmm_allocator_basic/grow_allocation is SKIPPED on all CI Windows runners #1241

Closed

Description

@rwgk

QA reported test_vmm_allocator_basic_allocation and test_vmm_allocator_grow_allocation failures (see below) under Windows.

Looking at the logs of the latest CI run for main:

It turns out our CI does not run those tests on any Windows machine.

What would it take to run the tests on at least one Windows machine?

Test failures reproduced interactively (TITAN RTX, WDDM):

(TestVenv) PS C:\Users\rgrossekunst\forked\ctk-next\cuda_core> pytest -ra -s -v .\tests\test_memory.py
========================================================================================= test session starts =========================================================================================
platform win32 -- Python 3.13.9, pytest-9.0.1, pluggy-1.6.0 -- C:\Users\rgrossekunst\forked\ctk-next\TestVenv\Scripts\python.exe
cachedir: .pytest_cache
benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: C:\Users\rgrossekunst\forked\ctk-next\cuda_core\tests
configfile: pytest.ini
plugins: benchmark-5.2.3
collected 37 items
...
============================================================================================== FAILURES ===============================================================================================
_________________________________________________________________________________ test_vmm_allocator_basic_allocation _________________________________________________________________________________

    def test_vmm_allocator_basic_allocation():
        """Test basic VMM allocation functionality.

        This test verifies that VirtualMemoryResource can allocate memory
        using CUDA VMM APIs with default configuration.
        """
        device = Device()
        device.set_current()

        # Skip if virtual memory management is not supported
        if not device.properties.virtual_memory_management_supported:
            pytest.skip("Virtual memory management is not supported on this device")

        options = VirtualMemoryResourceOptions()
        # Create VMM allocator with default config
        vmm_mr = VirtualMemoryResource(device, config=options)

        # Test basic allocation
>       buffer = vmm_mr.allocate(4096)
                 ^^^^^^^^^^^^^^^^^^^^^

tests\test_memory.py:333:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cuda\core\experimental\_memory\_virtual_memory_resource.py:516: in allocate
    raise_if_driver_error(res)
cuda\core\experimental\_utils\cuda_utils.pyx:67: in cuda.core.experimental._utils.cuda_utils._check_driver_error
    cpdef inline int _check_driver_error(cydriver.CUresult error) except?-1 nogil:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   raise CUDAError(f"{name.decode()}: {expl}")
E   cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.

cuda\core\experimental\_utils\cuda_utils.pyx:78: CUDAError
_________________________________________________________________________________ test_vmm_allocator_grow_allocation __________________________________________________________________________________

    def test_vmm_allocator_grow_allocation():
        """Test VMM allocator's ability to grow existing allocations.

        This test verifies that VirtualMemoryResource can grow existing
        allocations while preserving the base pointer when possible.
        """
        device = Device()
        device.set_current()

        # Skip if virtual memory management is not supported (we need it for VMM)
        if not device.properties.virtual_memory_management_supported:
            pytest.skip("Virtual memory management is not supported on this device")

        options = VirtualMemoryResourceOptions()

        vmm_mr = VirtualMemoryResource(device, config=options)

        # Create initial allocation
>       buffer = vmm_mr.allocate(2 * 1024 * 1024)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tests\test_memory.py:435:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cuda\core\experimental\_memory\_virtual_memory_resource.py:516: in allocate
    raise_if_driver_error(res)
cuda\core\experimental\_utils\cuda_utils.pyx:67: in cuda.core.experimental._utils.cuda_utils._check_driver_error
    cpdef inline int _check_driver_error(cydriver.CUresult error) except?-1 nogil:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   raise CUDAError(f"{name.decode()}: {expl}")
E   cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.

cuda\core\experimental\_utils\cuda_utils.pyx:78: CUDAError
======================================================================================= short test summary info =======================================================================================
SKIPPED [1] tests\test_memory.py:369: This test requires a device that doesn't support GPU Direct RDMA
SKIPPED [1] tests\test_memory.py:607: Driver rejects IPC-enabled mempool creation on this platform
FAILED tests\test_memory.py::test_vmm_allocator_basic_allocation - cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
FAILED tests\test_memory.py::test_vmm_allocator_grow_allocation - cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
=============================================================================== 2 failed, 33 passed, 2 skipped in 1.03s ===============================================================================

rwgk-win11.localdomain:~/logs_19339188192 $ grep test_vmm_allocator_basic_allocation *Test*.txt | grep -v SKIPPED | cut -d: -f1
Test_linux-64___py3.10__12.9.1__local__GPU_v100.txt
Test_linux-64___py3.10__13.0.2__wheels__GPU_l4.txt
Test_linux-64___py3.11__12.9.1__wheels__GPU_rtxpro6000.txt
Test_linux-64___py3.11__13.0.2__local__GPU_l4.txt
Test_linux-64___py3.12__12.9.1__local__GPU_l4.txt
Test_linux-64___py3.12__13.0.2__wheels__GPU_l4.txt
Test_linux-64___py3.13__12.9.1__wheels__GPU_v100.txt
Test_linux-64___py3.13__13.0.2__local__GPU_H100.txt
Test_linux-64___py3.13__13.0.2__local__GPU_rtxpro6000.txt
Test_linux-64___py3.14__13.0.2__local__GPU_l4.txt
Test_linux-64___py3.14t__13.0.2__local__GPU_l4.txt
Test_linux-aarch64___py3.10__12.9.1__local__GPU_a100.txt
Test_linux-aarch64___py3.10__13.0.2__wheels__GPU_a100.txt
Test_linux-aarch64___py3.11__12.9.1__wheels__GPU_a100.txt
Test_linux-aarch64___py3.11__13.0.2__local__GPU_a100.txt
Test_linux-aarch64___py3.12__12.9.1__local__GPU_a100.txt
Test_linux-aarch64___py3.12__13.0.2__wheels__GPU_a100.txt
Test_linux-aarch64___py3.13__12.9.1__wheels__GPU_a100.txt
Test_linux-aarch64___py3.13__13.0.2__local__GPU_a100.txt
Test_linux-aarch64___py3.14__13.0.2__local__GPU_a100.txt
Test_linux-aarch64___py3.14t__13.0.2__local__GPU_a100.txt
rwgk-win11.localdomain:~/logs_19339188192 $ grep test_vmm_allocator_basic_allocation *Test*.txt | grep SKIPPED | cut -d: -f1
Test_win-64___py3.12__12.9.1__local__GPU_t4.txt
Test_win-64___py3.12__12.9.1__wheels__GPU_l4.txt
Test_win-64___py3.13__13.0.2__local__GPU_l4.txt
Test_win-64___py3.13__13.0.2__wheels__GPU_t4.txt
Test_win-64___py3.14__13.0.2__local__GPU_l4.txt
Test_win-64___py3.14__13.0.2__wheels__GPU_t4.txt
Test_win-64___py3.14t__13.0.2__local__GPU_l4.txt
Test_win-64___py3.14t__13.0.2__wheels__GPU_t4.txt
rwgk-win11.localdomain:~/logs_19339188192 $ grep test_vmm_allocator_grow_allocation *Test*.txt | grep -v SKIPPED | cut -d: -f1
Test_linux-64___py3.10__12.9.1__local__GPU_v100.txt
Test_linux-64___py3.10__13.0.2__wheels__GPU_l4.txt
Test_linux-64___py3.11__12.9.1__wheels__GPU_rtxpro6000.txt
Test_linux-64___py3.11__13.0.2__local__GPU_l4.txt
Test_linux-64___py3.12__12.9.1__local__GPU_l4.txt
Test_linux-64___py3.12__13.0.2__wheels__GPU_l4.txt
Test_linux-64___py3.13__12.9.1__wheels__GPU_v100.txt
Test_linux-64___py3.13__13.0.2__local__GPU_H100.txt
Test_linux-64___py3.13__13.0.2__local__GPU_rtxpro6000.txt
Test_linux-64___py3.14__13.0.2__local__GPU_l4.txt
Test_linux-64___py3.14t__13.0.2__local__GPU_l4.txt
Test_linux-aarch64___py3.10__12.9.1__local__GPU_a100.txt
Test_linux-aarch64___py3.10__13.0.2__wheels__GPU_a100.txt
Test_linux-aarch64___py3.11__12.9.1__wheels__GPU_a100.txt
Test_linux-aarch64___py3.11__13.0.2__local__GPU_a100.txt
Test_linux-aarch64___py3.12__12.9.1__local__GPU_a100.txt
Test_linux-aarch64___py3.12__13.0.2__wheels__GPU_a100.txt
Test_linux-aarch64___py3.13__12.9.1__wheels__GPU_a100.txt
Test_linux-aarch64___py3.13__13.0.2__local__GPU_a100.txt
Test_linux-aarch64___py3.14__13.0.2__local__GPU_a100.txt
Test_linux-aarch64___py3.14t__13.0.2__local__GPU_a100.txt
rwgk-win11.localdomain:~/logs_19339188192 $ grep test_vmm_allocator_grow_allocation *Test*.txt | grep SKIPPED | cut -d: -f1
Test_win-64___py3.12__12.9.1__local__GPU_t4.txt
Test_win-64___py3.12__12.9.1__wheels__GPU_l4.txt
Test_win-64___py3.13__13.0.2__local__GPU_l4.txt
Test_win-64___py3.13__13.0.2__wheels__GPU_t4.txt
Test_win-64___py3.14__13.0.2__local__GPU_l4.txt
Test_win-64___py3.14__13.0.2__wheels__GPU_t4.txt
Test_win-64___py3.14t__13.0.2__local__GPU_l4.txt
Test_win-64___py3.14t__13.0.2__wheels__GPU_t4.txt

Activity

  1. added
    CI/CDCI/CD infrastructure
    testImprovements or additions to tests
    cuda.coreEverything related to the cuda.core module
    on Nov 15, 2025
  2. rwgk commented on Nov 15, 2025

    @rwgk
    ContributorAuthor

    @benhg for visibility

  3. rwgk commented on Nov 15, 2025

    @rwgk
    ContributorAuthor

    Quick look at the nvidia-smi outputs in the Windows logs:

    Currently we're only testing with TCC driver mode.

    (misc) rwgk-win11.localdomain:~/logs_19339188192 $ extract_nvidia-smi_outputs.py *Test*win*.txt | grep 00000000
    |   0  Tesla T4                     TCC   |   00000000:0A:00.0 Off |                    0 |
    |   0  NVIDIA L4                    TCC   |   00000000:0A:00.0 Off |                    0 |
    |   0  NVIDIA L4                    TCC   |   00000000:0A:00.0 Off |                    0 |
    |   0  Tesla T4                     TCC   |   00000000:0A:00.0 Off |                    0 |
    |   0  NVIDIA L4                    TCC   |   00000000:0A:00.0 Off |                    0 |
    |   0  Tesla T4                     TCC   |   00000000:0A:00.0 Off |                    0 |
    |   0  NVIDIA L4                    TCC   |   00000000:0A:00.0 Off |                    0 |
    |   0  Tesla T4                     TCC   |   00000000:0A:00.0 Off |                    0 |
    
  4. rwgk commented on Nov 16, 2025

    @rwgk
    ContributorAuthor

    @benhg @leofang

    The two tests (test_vmm_allocator_basic/grow_allocation) are failing with CTK 13.0.1 (released) already.

    The below is on my main workstation, which isn't special in any way.

    I believe the most important takeaway is that we need to ensure the cuda-python CI exercises at least one Windows machine with a driver in WDDM mode.

    Using cuda-python main as-is:

    (TestVenv) PS C:\Users\rgrossekunst\forked\cuda-python\cuda_core> git log -n 1
    commit 9aaa671dfd9699eb93f256f50d069999d38290f2 (HEAD -> main, upstream/main, upstream/HEAD)
    Author: Michael Droettboom <[email protected]>
    Date:   Thu Nov 13 11:52:13 2025 -0500
    
        Fix #1051: Make _graph.py compatible with cuda-python==12.6.* and fix tests (#1236)
    
    (TestVenv) PS C:\Users\rgrossekunst\forked\cuda-python\cuda_core> nvidia-smi
    Sat Nov 15 20:04:31 2025
    +-----------------------------------------------------------------------------------------+
    | NVIDIA-SMI 581.15                 Driver Version: 581.15         CUDA Version: 13.0     |
    +-----------------------------------------+------------------------+----------------------+
    | GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
    |                                         |                        |               MIG M. |
    |=========================================+========================+======================|
    |   0  NVIDIA RTX A6000             WDDM  |   00000000:C1:00.0 Off |                  Off |
    | 30%   33C    P8             20W /  300W |    1792MiB /  49140MiB |      1%      Default |
    |                                         |                        |                  N/A |
    +-----------------------------------------+------------------------+----------------------+
    
    (TestVenv) PS C:\Users\rgrossekunst\forked\cuda-python\cuda_core> pytest -ra -s -v .\tests\test_memory.py
    ============================================================ test session starts ============================================================
    platform win32 -- Python 3.13.9, pytest-9.0.1, pluggy-1.6.0 -- C:\Users\rgrossekunst\forked\cuda-python\TestVenv\Scripts\python.exe
    cachedir: .pytest_cache
    benchmark: 5.2.3 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
    rootdir: C:\Users\rgrossekunst\forked\cuda-python\cuda_core\tests
    configfile: pytest.ini
    plugins: benchmark-5.2.3
    collected 37 items
    
    tests\test_memory.py::test_package_contents PASSED
    tests\test_memory.py::test_buffer_initialization PASSED
    tests\test_memory.py::test_buffer_copy_to PASSED
    tests\test_memory.py::test_buffer_copy_from PASSED
    tests\test_memory.py::test_buffer_close PASSED
    tests\test_memory.py::test_buffer_dunder_dlpack PASSED
    tests\test_memory.py::test_buffer_dunder_dlpack_device_success[DummyDeviceMemoryResource-expected0] PASSED
    tests\test_memory.py::test_buffer_dunder_dlpack_device_success[DummyHostMemoryResource-expected1] PASSED
    tests\test_memory.py::test_buffer_dunder_dlpack_device_success[DummyUnifiedMemoryResource-expected2] PASSED
    tests\test_memory.py::test_buffer_dunder_dlpack_device_success[DummyPinnedMemoryResource-expected3] PASSED
    tests\test_memory.py::test_buffer_dunder_dlpack_device_failure PASSED
    tests\test_memory.py::test_device_memory_resource_initialization[True] PASSED
    tests\test_memory.py::test_device_memory_resource_initialization[False] PASSED
    tests\test_memory.py::test_vmm_allocator_basic_allocation FAILED
    tests\test_memory.py::test_vmm_allocator_policy_configuration SKIPPED (This test requires a device that doesn't support GPU Direct...)
    tests\test_memory.py::test_vmm_allocator_grow_allocation FAILED
    tests\test_memory.py::test_vmm_allocator_rdma_unsupported_exception PASSED
    tests\test_memory.py::test_mempool PASSED
    tests\test_memory.py::test_mempool_attributes[reuse_follow_event_dependencies-bool-True] PASSED
    tests\test_memory.py::test_mempool_attributes[reuse_follow_event_dependencies-bool-False] PASSED
    tests\test_memory.py::test_mempool_attributes[reuse_allow_opportunistic-bool-True] PASSED
    tests\test_memory.py::test_mempool_attributes[reuse_allow_opportunistic-bool-False] PASSED
    tests\test_memory.py::test_mempool_attributes[reuse_allow_internal_dependencies-bool-True] PASSED
    tests\test_memory.py::test_mempool_attributes[reuse_allow_internal_dependencies-bool-False] PASSED
    tests\test_memory.py::test_mempool_attributes[release_threshold-int-True] PASSED
    tests\test_memory.py::test_mempool_attributes[release_threshold-int-False] PASSED
    tests\test_memory.py::test_mempool_attributes[reserved_mem_current-int-True] PASSED
    tests\test_memory.py::test_mempool_attributes[reserved_mem_current-int-False] PASSED
    tests\test_memory.py::test_mempool_attributes[reserved_mem_high-int-True] PASSED
    tests\test_memory.py::test_mempool_attributes[reserved_mem_high-int-False] PASSED
    tests\test_memory.py::test_mempool_attributes[used_mem_current-int-True] PASSED
    tests\test_memory.py::test_mempool_attributes[used_mem_current-int-False] PASSED
    tests\test_memory.py::test_mempool_attributes[used_mem_high-int-True] PASSED
    tests\test_memory.py::test_mempool_attributes[used_mem_high-int-False] PASSED
    tests\test_memory.py::test_mempool_attributes_ownership SKIPPED (Driver rejects IPC-enabled mempool creation on this platform)
    tests\test_memory.py::test_strided_memory_view_leak PASSED
    tests\test_memory.py::test_strided_memory_view_refcnt PASSED
    
    ================================================================= FAILURES ==================================================================
    ____________________________________________________ test_vmm_allocator_basic_allocation ____________________________________________________
    
        def test_vmm_allocator_basic_allocation():
            """Test basic VMM allocation functionality.
    
            This test verifies that VirtualMemoryResource can allocate memory
            using CUDA VMM APIs with default configuration.
            """
            device = Device()
            device.set_current()
    
            # Skip if virtual memory management is not supported
            if not device.properties.virtual_memory_management_supported:
                pytest.skip("Virtual memory management is not supported on this device")
    
            options = VirtualMemoryResourceOptions()
            # Create VMM allocator with default config
            vmm_mr = VirtualMemoryResource(device, config=options)
    
            # Test basic allocation
    >       buffer = vmm_mr.allocate(4096)
                     ^^^^^^^^^^^^^^^^^^^^^
    
    tests\test_memory.py:333:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    cuda\core\experimental\_memory\_virtual_memory_resource.py:516: in allocate
        raise_if_driver_error(res)
    cuda\core\experimental\_utils\cuda_utils.pyx:67: in cuda.core.experimental._utils.cuda_utils._check_driver_error
        cpdef inline int _check_driver_error(cydriver.CUresult error) except?-1 nogil:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    >   raise CUDAError(f"{name.decode()}: {expl}")
    E   cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
    
    cuda\core\experimental\_utils\cuda_utils.pyx:78: CUDAError
    ____________________________________________________ test_vmm_allocator_grow_allocation _____________________________________________________
    
        def test_vmm_allocator_grow_allocation():
            """Test VMM allocator's ability to grow existing allocations.
    
            This test verifies that VirtualMemoryResource can grow existing
            allocations while preserving the base pointer when possible.
            """
            device = Device()
            device.set_current()
    
            # Skip if virtual memory management is not supported (we need it for VMM)
            if not device.properties.virtual_memory_management_supported:
                pytest.skip("Virtual memory management is not supported on this device")
    
            options = VirtualMemoryResourceOptions()
    
            vmm_mr = VirtualMemoryResource(device, config=options)
    
            # Create initial allocation
    >       buffer = vmm_mr.allocate(2 * 1024 * 1024)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    
    tests\test_memory.py:435:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    cuda\core\experimental\_memory\_virtual_memory_resource.py:516: in allocate
        raise_if_driver_error(res)
    cuda\core\experimental\_utils\cuda_utils.pyx:67: in cuda.core.experimental._utils.cuda_utils._check_driver_error
        cpdef inline int _check_driver_error(cydriver.CUresult error) except?-1 nogil:
    _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
    
    >   raise CUDAError(f"{name.decode()}: {expl}")
    E   cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.
    
    cuda\core\experimental\_utils\cuda_utils.pyx:78: CUDAError
    ========================================================== short test summary info ==========================================================
    SKIPPED [1] tests\test_memory.py:369: This test requires a device that doesn't support GPU Direct RDMA
    SKIPPED [1] tests\test_memory.py:607: Driver rejects IPC-enabled mempool creation on this platform
    FAILED tests\test_memory.py::test_vmm_allocator_basic_allocation - cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to...
    FAILED tests\test_memory.py::test_vmm_allocator_grow_allocation - cuda.core.experimental._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to...
    ================================================== 2 failed, 33 passed, 2 skipped in 0.36s ==================================================
    
  5. leofang commented on Nov 16, 2025

    @leofang
    Member

    This is a duplicate of #985. I think I've said this multiple times during team meetings that it's a gap that we're closing :)

    Currently we're only testing with TCC driver mode.

    I believe the most important takeaway is that we need to ensure the cuda-python CI exercises at least one Windows machine with a driver in WDDM mode.

    All of these are known and nothing is new.

  6. rwgk commented on Nov 16, 2025

    @rwgk
    ContributorAuthor

    All of these are known and nothing is new.

    Please forgive me that I cannot absorb all those things all at once.

  7. leofang commented on Nov 16, 2025

    @leofang
    Member

    xref #462, #984

  8. leofang commented on Nov 17, 2025

    @leofang
    Member

    With #1242 the 2 VMM errors can be consistently reproduced with MCDM modes too (we are pending the runner team's approval for granting access to gamer cards for testing WDDM modes, see https://github.com/nv-gha-runners/enterprise-runner-configuration/pull/219).

  9. leofang commented on Nov 18, 2025

    @leofang
    Member

    (we are pending the runner team's approval for granting access to gamer cards for testing WDDM modes, see nv-gha-runners/enterprise-runner-configuration#219).

    Update: we got runner access and this now reproduces on both WDDM and MCDM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CI/CDCI/CD infrastructurecuda.coreEverything related to the cuda.core moduletestImprovements or additions to teststriageNeeds the team's attention

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions