[BUG]: CUDA API calls through cuda-bindings 3x slower than direct CUDA C++ API calls

Is this a duplicate?

I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Performance

Component

cuda.bindings

Describe the bug

I have a same CUDA kernel, and I am calling it using two different methods.

Compile the kernel to .cubin, load it in Python and call it using cuda-bindings. The TMA descriptors are also created via cuda-bindings.
Compile the kernel and relevant C++ host functions (including TMA desc, kernel attribute and launch config) to .so, load the .so in Python and call the C++ host function.

As revealed in the two timelines below, the first path is 3x slower than the second path.

How to Reproduce

Follow the instructions in the reproduction repo https://github.com/lucifer1004/cuda-python-repro

Expected behavior

CUDA Python's perf should be on par with CUDA C++.

Operating System

Ubuntu Linux 24.04

nvidia-smi output

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+

@leofang

@leofang

Maybe related. I have a cuda kernel that seems to run slow if i compile it to cubin and run it in the same python run even with a round of warmups. However after i cache the cubin to a database and run it a second time from a second python run it's much faster.

If this is odd and unexpected behavior I can paste a link to the code that's doing this.

@capybara-club

@capybara-club could you kindly create a new issue with a reproducer so that we can look at? It is unlikely that we can help you based only on this comment 😅

@lucifer1004 sorry this fell through the cracks. A few comments

your kernel.py allocates 3 TMA descriptors, set kernel attribute, and then launch a kernel
however, your wrapped_kernel.cu moves the kernel attribute setting out of the timing loop (here)
also, the way kernel.py is written adds obvious getattr overheads (very unnatural to Python eyes 😅): Instead of

import cuda.bindings.driver

cuda.bindings.driver.A(...)
cuda.bindings.driver.B(...)
cuda.bindings.driver.C(...)

please rewrite it to import what you need

from cuda.bindings import driver

driver.A(...)
driver.B(...)
driver.C(...)

or even better

from cuda.bindings.driver import A, B, C

A(...)
B(...)
C(...)

@mdboom

cc @mdboom for vis

@leofang

@leofang I have simplified the code to only measure TMA descriptor creation. And here are the updated results:

CUDA 13.0

PIXI_ENV=cu13 just build benchmark
pixi run -e cu13 nvcc -gencode=arch=compute_90a,code=sm_90a -O3 -shared -Xcompiler -fPIC -lcuda -o libwrapped_kernel.so wrapped_kernel.cu
nvcc warning : incompatible redefinition for option 'compiler-bindir', the last value of this option was used
pixi run -e cu13 python main.py
TMA Descriptor Creation Benchmark: Python vs C++
======================================================================
Using CUDA device: NVIDIA H100 80GB HBM3
Python version: 3.14.0
nvcc warning : incompatible redefinition for option 'compiler-bindir', the last value of this option was used
NVCC version: cuda_13.0.r13.0/compiler.36424714_0
CUDA Python version: 13.0.3

Benchmarking Python TMA descriptor creation (1000 iterations)...
Python - Total: 0.005918s, Average: 5.918μs per iteration

Benchmarking C++ TMA descriptor creation (1000 iterations)...
C++    - Total: 0.000157s, Average: 0.157μs per iteration

Performance Comparison:
------------------------------
C++ is 37.76x faster than Python
Time difference: 5.761μs per iteration

CUDA 12.9

PIXI_ENV=cu12 just build benchmark
pixi run -e cu12 nvcc -gencode=arch=compute_90a,code=sm_90a -O3 -shared -Xcompiler -fPIC -lcuda -o libwrapped_kernel.so wrapped_kernel.cu
nvcc warning : incompatible redefinition for option 'compiler-bindir', the last value of this option was used
pixi run -e cu12 python main.py
TMA Descriptor Creation Benchmark: Python vs C++
======================================================================
Using CUDA device: NVIDIA H100 80GB HBM3
Python version: 3.14.0
nvcc warning : incompatible redefinition for option 'compiler-bindir', the last value of this option was used
NVCC version: cuda_12.9.r12.9/compiler.36037853_0
CUDA Python version: 12.9.4

Benchmarking Python TMA descriptor creation (1000 iterations)...
Python - Total: 0.005843s, Average: 5.843μs per iteration

Benchmarking C++ TMA descriptor creation (1000 iterations)...
C++    - Total: 0.000153s, Average: 0.153μs per iteration

Performance Comparison:
------------------------------
C++ is 38.30x faster than Python
Time difference: 5.690μs per iteration

I thought I'd report back on some findings. I have had a hard time getting access to hardware that can run the reproducer, however, it is easy enough to comment out the actual CUDA call and just measure the Python calling overhead on its own. That's good enough to find improvements while there is a lot of low-hanging fruit to be had, but eventually I'll probably need to reproduce more accurately. Just be aware that /my/ numbers don't include any time doing actual work on the device.

Accessing values of enums

It turns out that instead of

cdef cydriver.CUtensorMapInterleave cyinterleave = interleave.value

doing this:

cdef cydriver.CUtensorMapInterleave cyinterleave = int(interleave)

reduces the function call overhead by about 25%

This is because IntEnum inherits from int, so there is a good fast path in C to grab its value quickly this way. enum.value has to do both a dictionary lookup for value and run the Python-written value property code.

This feels like a simple win that will be easy to implement.

Converting short lists/tuples of `driver.cuuint32_t` values

This function has 4 arguments which take short lists or tuples of uint32_t or uint64_t.

    elementStrides = [] if elementStrides is None else elementStrides
    if not all(isinstance(_x, (cuuint32_t,)) for _x in elementStrides):
        raise TypeError("Argument 'elementStrides' is not instance of type (expected tuple[cydriver.cuuint32_t,] or list[cydriver.cuuint32_t,]")

    ...

    cdef cydriver.cuuint32_t* cyelementStrides = NULL
    if len(elementStrides) > 1:
        cyelementStrides = <cydriver.cuuint32_t*> calloc(len(elementStrides), sizeof(cydriver.cuuint32_t))
        if cyelementStrides is NULL:
            raise MemoryError('Failed to allocate length x size memory: ' + str(len(elementStrides)) + 'x' + str(sizeof(cydriver.cuuint32_t)))
        else:
            for idx in range(len(elementStrides)):
                cyelementStrides[idx] = <cydriver.cuuint32_t>(<cuuint32_t>elementStrides[idx])._pvt_ptr[0]
    elif len(elementStrides) == 1:
        cyelementStrides = <cydriver.cuuint32_t*>(<cuuint32_t>elementStrides[0])._pvt_ptr

    ...  # Make the CUDA call

    if len(elementStrides) > 1 and cyelementStrides is not NULL:
        free(cyelementStrides)

The first performance problem is the type checking happens inside of a generator expression:

if not all(isinstance(_x, (cuuint32_t,)) for _x in elementStrides):
        raise TypeError("Argument 'elementStrides' is not instance of type (expected tuple[cydriver.cuuint32_t,] or list[cydriver.cuuint32_t,]")

This generates a separate Python extension function to handle the generator callback which is then handed to Python's all function. This is significantly more expensive than a C for loop. Additionally, this uses isinstance, when an exact type comparison would do: type(_x) is cuuint32_t. Changing this to a for loop and a direct type check reduces overhead by another 10%:

for x_ in elementStrides:
    if type(_x) is not cuuint32_t:
        raise TypeError(...)

The next thing we can tackle is that when the length of the list is > 1, it dynamically allocates a buffer on the heap to copy raw integer values into, and frees it at the end. This API (and I /think/ all CUDA APIs) have a maximum dimensionality of 5, so we can just statically allocate an array of 5 on the stack, avoiding dynamic memory allocation. This should be faster and also eliminate the latent memory leaks in the current implementation.

All told, the solution I came up with, which also leans on Cython to do the type-checking implicitly with a ? cast, gives another 25% speedup:

    cdef cydriver.cuuint32_t* cyelementStrides
    cdef unsigned int elementStridesLen
    cdef cydriver.cuuint32_t[5] elementStridesStatic
    elementStridesLen = 0 if elementStrides is None else len(elementStrides)
    if elementStridesLen == 0:
        cyelementStrides = NULL
    elif elementStridesLen == 1:
        cyelementStrides = <cydriver.cuuint32_t*>(<cuuint32_t>elementStrides[0])._pvt_ptr
    elif elementStridesLen < 6:
        for idx in range(elementStridesLen):
            elementStridesStatic[idx] = <cydriver.cuuint32_t>(<cuuint32_t?>elementStrides[idx])._pvt_ptr[0]
        cyelementStrides = elementStridesStatic
    else:
        raise ValueError("elementStrides argument too long, must be <= 5")

This seems like a manageable change. The part that makes it a bit tricky in the generator is that we can only do this static allocation trick for arguments that are small sequences representing dimensions -- it wouldn't work in the general case of any list of ints. So we may need to add metadata about these arguments where this optimization can be applied.

Using the `Py_buffer` interface (rejected)

I also experimented with using the Py_buffer interface for these small arrays. The idea being that rather than passing in a list of special uint32_t objects, you would use something exposing the buffer interface like a stdlib array.array or Numpy array. Theoretically, it could be faster because all elements are guaranteed to be the same (so you don't have to do a linear scan to typecheck), and in the common case it boils down to a raw pointer to contiguous memory so no copying is required. I implemented this and saw no measurable difference from the solution above, so it is definitely not worth that backward-incompatibility downside. Were these arrays larger, we might see benefit, but with these < 5 element things it seems to be a wash.

This API (and I /think/ all CUDA APIs) have a maximum dimensionality of 5, so we can just statically allocate an array of 5 on the stack, avoiding dynamic memory allocation.

Yes, this makes sense because TMA currently only supports up to 5D. FWIW, Cython's default for memoryview is 8D.

@lucifer1004

@lucifer1004: With the 2 low-hanging fruit fixes merged, it would be great to hear how this affects your measurements on your hardware when you have a chance.

Is there nightly build I can use for benchmarking?

Is there nightly build I can use for benchmarking?

Sorry for the delayed response. You can download a recent build from any of our CI actions here: https://github.com/NVIDIA/cuda-python/actions/workflows/ci.yml

Closing this issue since the majority of perf improvement work is already done. See the detailed breakdown (and remaining tasks) in #1645.

Navigation Menu

Repository navigation

[BUG]: CUDA API calls through `cuda-bindings` 3x slower than direct CUDA C++ API calls #659

Is this a duplicate?

Type of Bug

Component

Describe the bug

How to Reproduce

Expected behavior

Operating System

nvidia-smi output

Accessing values of enums

Converting short lists/tuples of `driver.cuuint32_t` values

Using the `Py_buffer` interface (rejected)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[BUG]: CUDA API calls through cuda-bindings 3x slower than direct CUDA C++ API calls #659

Description

Is this a duplicate?

Type of Bug

Component

Describe the bug

How to Reproduce

Expected behavior

Operating System

nvidia-smi output

Activity

capybara-club commented on Jun 5, 2025

leofang commented on Sep 10, 2025

leofang commented on Sep 10, 2025

lucifer1004 commented on Oct 28, 2025

mdboom commented on Jan 29, 2026

Accessing values of enums

Converting short lists/tuples of driver.cuuint32_t values

Using the Py_buffer interface (rejected)

leofang commented on Feb 2, 2026

mdboom commented on Feb 5, 2026

lucifer1004 commented on Feb 7, 2026

mdboom commented on Feb 18, 2026

leofang commented on Apr 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[BUG]: CUDA API calls through `cuda-bindings` 3x slower than direct CUDA C++ API calls #659

Converting short lists/tuples of `driver.cuuint32_t` values

Using the `Py_buffer` interface (rejected)