Repository navigation
[FEA]: Trigger add_dll_directory(found_path) logic even if was_already_loaded_from_elsewhere is true #821
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct
Type of Bug
Runtime Error
Component
cuda.bindings
Describe the bug
Attempting to use NVRTC on Windows with the NVRTC wheels from cuda-python fails with:
nvrtc: error: failed to open nvrtc-builtins64_129.dll.
Make sure that nvrtc-builtins64_129.dll is installed correctly.
I can work around this by copying nvrtc-builtins64_129.dll to a folder where other loaded DLLs exist, e.g.
copy "$SYS_PREFIX\Lib\site-packages\nvidia\cuda_nvrtc\bin\nvrtc-builtins64_*.dll" "$SYS_PREFIX"How to Reproduce
I will have to create a smaller reproducer, but I observed this by pip installing Numba-CUDA in CI and running its testsuite: https://github.com/NVIDIA/numba-cuda/pull/353/files#diff-9753d897094baa6cf320a53ee98e7059f8477dbf3c6d467078ce35bcf3354f57R31-R57
Example run: https://github.com/NVIDIA/numba-cuda/actions/runs/16526296179/job/46740959886#step:8:8351
I did similar on my local machine and observed the same issue. I will update this issue once I have a shorter reproducer.
Expected behavior
Using NVRTC should not emit an NVRTC error about loading the builtins.
Operating System
Windows 11, amd64
nvidia-smi output
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 572.13 Driver Version: 572.13 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Driver-Model | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 Tesla T4 TCC | 00000001:00:00.0 Off | Off |
| N/A 38C P8 9W / 70W | 9MiB / 16384MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Metadata
Metadata
Assignees
Type
Projects
- Status
Activity
gmarkall commentedon Aug 8, 2025
As discussed in Slack, this is probably because Numba-CUDA has a couple of routes by which it opens NVRTC itself, but then attempts to use the CUDA bindings to actually compile programs.
leofang commentedon Aug 8, 2025
This is how the NVRTC wheel is supported in
cuda-bindings/cuda-core:cuda-bindings12.8 and earlier, we preload nvrtc-builtins when loading nvrtc here (I applied the same logic from nvmath)add_dll_directorysupport to generalize this logic without hard-coding the dll namesNow to expand what Graham noted above:
add_dll_directorylogic is part of the search logic, so in this case the nvrtc-builtins dll is not loaded or placed in the search pathsThis seems like a very sharp edge in pathfinder, for which I am not sure if it's fixable at all. In an offline discussion, @rwgk suggested:
gmarkall commentedon Aug 8, 2025
Numba-cuda should not be loading nvrtc at all when using the cuda bindings, so if I fix that problem then I think we should avoid this sharp edge?
leofang commentedon Aug 8, 2025
Yes that'd be ideal.
leofang commentedon Aug 8, 2025
I keep this issue open for now (until Ralf makes a decision) just because I have a feeling that numba-cuda might not be the only one that gets trapped by this.
[-][BUG]: Attempting to use NVRTC on Windows with the NVRTC wheels from cuda-python fails to load NVRTC builtins[/-][+][cuda_pathfinder][FEAT]: Trigger `add_dll_directory(found_path)` logic even if `was_already_loaded_from_elsewhere` is true[/+][-][cuda_pathfinder][FEAT]: Trigger `add_dll_directory(found_path)` logic even if `was_already_loaded_from_elsewhere` is true[/-][+][FEA]: Trigger `add_dll_directory(found_path)` logic even if `was_already_loaded_from_elsewhere` is true[/+]carterbox commentedon Aug 15, 2025
I think nvmath-python is also catching this issue in our attempted migration to pathfinder. We tried to resolve this by calling
pathfinder.load_nvidia_dynamic_lib('nvrtc-builtins'), but I believe this doesn't work becausenvrtc-builtinsis not inSUPPORTED_NVIDIA_LIBNAMES.Would supporting loading
nvrtc-builtinsseparately fromnvrtcsolve the issue?leofang commentedon Aug 15, 2025
@carterbox it must be a bug in your environment and it is hard to triage for you without further information. NVRTC is properly supported as explained above. You don't need to additionally load nvrtc-builtin yourself.
ZzEeKkAa commentedon Aug 18, 2025
@leofang @carterbox I was able to reproduce the issue with
nvrtc-builtinsand filed a separate issue with all details: #849