Repository navigation
Local failures in test_green_context.py tests #2025
Description
Is there something I need to do to get these to pass?
FAILED cuda_core/tests/test_green_context.py::TestSMResourceSplit::test_two_groups - cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION: This error indicates one or more resources are insufficient or non-applicable ...
FAILED cuda_core/tests/test_green_context.py::TestGreenContextKernelLaunch::test_two_green_contexts_independent - cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION: This error indicates one or more resources are insufficient or non-applicable ...
FAILED cuda_core/tests/test_green_context.py::TestContextResources::test_green_ctx_resources_reflect_partition - cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION: This error indicates one or more resources are insufficient or non-applicable ...
________________________________________________________________ TestSMResourceSplit.test_two_groups ________________________________________________________________
self = <test_green_context.TestSMResourceSplit object at 0x71f99d592060>, sm_resource = <cuda.core._device_resources.SMResource object at 0x71f9786ad7c0>
def test_two_groups(self, sm_resource):
"""Two-group split with explicit aligned counts.""" half = _aligned_half(sm_resource)
if half < sm_resource.min_partition_size:
pytest.skip("Not enough SMs for a 2-group split")
> groups, rem = sm_resource.split(SMResourceOptions(count=(half, half)))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
half = 24
self = <test_green_context.TestSMResourceSplit object at 0x71f99d592060>
sm_resource = <cuda.core._device_resources.SMResource object at 0x71f9786ad7c0>
cuda_core/tests/test_green_context.py:246:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
cuda/core/_device_resources.pyx:475: in cuda.core._device_resources.SMResource.split
??? cuda/core/_device_resources.pyx:286: in cuda.core._device_resources._split_with_general_api
??? cuda/core/_utils/cuda_utils.pyx:65: in cuda.core._utils.cuda_utils.HANDLE_RETURN
??? _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> ???
E cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_RESOURCE_CONFIGURATION: This error indicates one or more resources are insufficient or non-applicable f
or the operation.
cuda/core/_utils/cuda_utils.pyx:163: CUDAError
Activity
leofang commentedon May 6, 2026
Lowering this to P1 as hitting
CUDA_ERROR_INVALID_RESOURCE_CONFIGURATIONis actually expected by the driver design. Raised #2031 to improve the situation. Will add my understanding here before closing the issue.Add backfill support to SMResourceOptions and fix topology-sensitive …
Add backfill support to SMResourceOptions and fix topology-sensitive …
leofang commentedon May 6, 2026
Understanding SM split constraints and the
backfilloptionWhen using
SMResource.split()to partition SM resources for green contexts, users may encounterCUDA_ERROR_INVALID_RESOURCE_CONFIGURATIONeven when the requested SM count is properly aligned tocoscheduled_alignment. This is expected behavior — the CUDA driver's structured split API (cuDevSmResourceSplit) enforces hardware topology constraints beyond simple alignment.What we found
coscheduled_alignment.The fix —
backfill=TrueSMResourceOptionsnow supports abackfillfield. When set toTrue, the driver relaxes the co-scheduling constraint and allows arbitrary aligned SM counts:The trade-off: with backfill, some SMs in each group may not have the co-scheduling guarantee (relevant for workloads using thread block clusters). For workloads that don't rely on clusters,
backfill=Trueis the recommended option for maximum flexibility.Alternatively, setting
coscheduled_sm_count=2removes the cluster co-scheduling requirement entirely, making every aligned-to-2 count valid without backfill.See the CUDA driver API documentation for details on
CU_DEV_SM_RESOURCE_GROUP_BACKFILLand the structured split API.Fix: #2031
-- Leo's bot