Repository navigation
RFC: Cythonize cuda.core while keeping it CUDA-agnostic #866
Description
Today the majority of cuda.core is implemented in pure Python. As a result, we've been dealing with microsecond-level overhead in the past few months if not weeks (ex: #739, #658). As much as I think it is pre-mature optimization at this stage, I do hear the desire of keeping the performance competitive while staying productive.
This RFC outlines one such solution to address the performance concerns. Below are the critical requirements
cuda.corecontinues to support multiple CUDA major versions- The installation UX
pip install cuda-corestays unchanged - Local development workflow is uninterrupted
- No user- or developer- visible breaking change is introduced (despite we're still in the experimental phase)
The critical question to answer is how we'll lower to Cython while having to build against cuda.bindings 12x & 13.x. Here is the steps following the great work @dalcinl did for mpi4py v4.1.0 (to support both Open MPI and MPICH):
- We turn all Python modules from
.pyto.pyxand update the build system- We could consider having one mega
.pyxwith others being.pyiand literal-included, similar to thempi4py.MPImodule
- We could consider having one mega
- We build
cuda-coretwice, once against CUDA &cuda-bindings12.x, and then 13.x - We merge two generated wheels into a single one (script)
- A runtime dispatching snippet should be injected into
cuda/core/experimental/__init__.pyto decide which extension module to load, based on the installedcuda-bindingsmajor version
- A runtime dispatching snippet should be injected into
It is worth noting that the Step 2 and 3 only happen in the public CI, so as to meet the Requirement 3 (for local development, neither internal nor external developers should need to have multiple CUDA versions installed).
Another note is that this RFC is only applicable to make our Python wheels stay variant-free (no -cu12/-cu13); for conda packages, it is trivial to build variant packages without changing the UX (conda install cuda-core), so no extra work is needed.
This RFC also mirrors our plan for cuda-cccl (NVIDIA/cccl#2555).
Metadata
Metadata
Assignees
Labels
Type
Projects
- Status
Activity
leofang commentedon Aug 20, 2025
cc @mdboom @shwina @carterbox @samaid @cryos @oleksandr-pavlyk @ZzEeKkAa
mdboom commentedon Aug 21, 2025
Some questions to help me catch up: Is the main source of overhead currently that
cuda.corecalls to things likecuda.bindings.driveras Python (so things are loosely-binary-coupled)? Or is there overhead in actually running Python code incuda.coreoutside of the more expensive function call? (With things like function inlining coming to Python 3.15, it may actually be faster to stay in Python, but it will be 2029 before that would be mainstream, so we have to ignore that for now...)Have we confirmed that this will help enough? In a lot of cases, calling functions across Cython modules still uses the Python function call machinery, so while probably faster than pure Python, it may still be significantly slower than a C function call. Or is the plan to call CUDA C++ directly from CUDA core after migrating to
.pyx? It might make sense to do this conversion for a single CUDA version (without dealing with any of the multiple version complexity) first to confirm that this will be sufficiently helpful. (Maybe that's already been confirmed...)carterbox commentedon Aug 26, 2025
I agree with Michael.
leofang commentedon Sep 27, 2025
Sorry, for some reason this fell through the cracks. I'll try to work this out before code freeze. Here's a short reply.
We have done perf analysis in #677 and the linked issues therein. Consider #677 as the SOL implementation: Instead of accessing the CUDA APIs through Python, it switched to access them through Cython (same as what we did in CuPy, the golden standard we used for latency benchmarks). The price it paid was to tying
cuda.coreto a specific CUDA major version due thecuda.bindingstransitive dependency on CUDA headers. It was deemed unacceptable so we switched to #709 instead, which allows us to shield the ABI requirements through Python import system and still support two CUDA major versions.This RFC is to set up the playground so that we can revisit #677.
Daniel you witnessed and participated in the entire discussion on this, what are you agreeing with exactly? You knew the context better than Mike or anyone else new to the team... lol
leofang commentedon Sep 27, 2025
FWIW, Ashwin was one step ahead of me and completed the wheel merge for
cuda-cccl: NVIDIA/cccl#5613, so in addition tompi4pywe have a second data point proving it's doable and meets expectations._context.pyxmore #1075_graph.py#1076_launch_config.py#1077_launcher.pyxmore #1078_linker.py#1079_memory.pyxmore #1080_module.py#1081_program.py#1082leofang commentedon Apr 7, 2026
Closing since most subtasks are done; the open ones are tracked separately.