Repository navigation
[BUG]: nvrtc-builtins failed to load on windows and system install on linux #849
Closed as not planned
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct
Type of Bug
Runtime Error
Component
cuda.pathfinder
Describe the bug
Failed to load nvrtc-builtins from system install on windows. On linux works fine. nvrtc works so:
(venv) PS C:\Users\yhavrylko> pip install cuda-pathfinder
Collecting cuda-pathfinder
Downloading cuda_pathfinder-1.1.0-py3-none-any.whl.metadata (2.3 kB)
Downloading cuda_pathfinder-1.1.0-py3-none-any.whl (17 kB)
Installing collected packages: cuda-pathfinder
Successfully installed cuda-pathfinder-1.1.0
(venv) PS C:\Users\yhavrylko> python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc'))"
LoadedDL(abs_path='C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v12.9\\bin\\nvrtc64_120_0.dll', was_already_loaded_from_elsewhere=False, _handle_uint=140730218643456)
(venv) PS C:\Users\yhavrylko> python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc-builtins'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc-builtins'))
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "C:\Users\yhavrylko\venv\Lib\site-packages\cuda\pathfinder\_dynamic_libs\load_nvidia_dynamic_lib.py", line 71, in load_nvidia_dynamic_lib
return _load_lib_no_cache(libname)
File "C:\Users\yhavrylko\venv\Lib\site-packages\cuda\pathfinder\_dynamic_libs\load_nvidia_dynamic_lib.py", line 44, in _load_lib_no_cache
found.raise_if_abs_path_is_None()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "C:\Users\yhavrylko\venv\Lib\site-packages\cuda\pathfinder\_dynamic_libs\find_nvidia_dynamic_lib.py", line 194, in raise_if_abs_path_is_None
raise DynamicLibNotFoundError(f'Failure finding "{self.lib_searched_for}": {err}\n{att}')
cuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError: Failure finding "nvrtc-builtins*.dll": No such file: nvrtc-builtins*.dll, No such file: nvrtc-builtins*.dll
listdir("C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.9\bin"):
__nvcc_device_query.exe
bin2c.exe
compute-sanitizer.bat
crt
cu++filt.exe
cublas64_12.dll
cublasLt64_12.dll
cudafe++.exe
cudart64_12.dll
cufft64_11.dll
cufftw64_11.dll
cuinj64_129.dll
cuobjdump.exe
curand64_10.dll
cusolver64_11.dll
cusolverMg64_11.dll
cusparse64_12.dll
fatbinary.exe
nppc64_12.dll
nppial64_12.dll
nppicc64_12.dll
nppidei64_12.dll
nppif64_12.dll
nppig64_12.dll
nppim64_12.dll
nppist64_12.dll
nppisu64_12.dll
nppitc64_12.dll
npps64_12.dll
nvJitLink_120_0.dll
nvblas64_12.dll
nvcc.exe
nvcc.profile
nvdisasm.exe
nvfatbin_120_0.dll
nvjpeg64_12.dll
nvlink.exe
nvprof.exe
nvprune.exe
nvrtc-builtins64_129.dll
nvrtc64_120_0.alt.dll
nvrtc64_120_0.dll
nvvp.bat
ptxas.exe
How to Reproduce
- Clean Windows install with latest nvidia driver, vs-redist C++ build tools 2022, ctk 12.9.1.
- Create python virtual env and activate it
pip install cuda-pathfinderpython -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc-builtins'))"
Expected behavior
Same as on linux, where it manages to find the library.
Operating System
Windows 11 24H
nvidia-smi output
n/a
Update
On linux it also fails to find sysctk install:
python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc-builtins'))"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 71, in load_nvidia_dynamic_lib
return _load_lib_no_cache(libname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 40, in _load_lib_no_cache
loaded = load_with_system_search(libname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 104, in load_with_system_search
abs_path = abs_path_for_dynamic_library(libname, handle)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 47, in abs_path_for_dynamic_library
for symbol_name in EXPECTED_LIB_SYMBOLS[libname]:
~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'nvrtc-builtins'
( most likely related to #833 )
However it is able to find with nvidia-cuda-nvrtc-cu12:
python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('nvrtc-builtins'))"
LoadedDL(abs_path='/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv/lib/python3.11/site-packages/nvidia/cuda_nvrtc/lib/libnvrtc-builtins.so.12.9', was_already_loaded_from_elsewhere=False, _handle_uint=94212168521552)
On windows it still does not work with pip install nvidia-cuda-nvrtc-cu12. (nvrtc works so)
Activity
[-][BUG]: nvrtc-builtins failed to load on windows[/-][+][BUG]: nvrtc-builtins failed to load on windows and system install on linux[/+]ZzEeKkAa commentedon Aug 18, 2025
There is
nvrtc-builtins64_129.dllin the list. Could it be because suffix is129, not12?leofang commentedon Aug 18, 2025
Sorry, but why is this a bug? I already said that
load_nvidia_dynamic_lib('nvrtc-builtins')is not needed (and in fact is not meant to be supported by the pathfinder). We just needload_nvidia_dynamic_lib('nvrtc')and that's all!Look at the pathfinder output, it does know where it is
and it's NVRTC's job to load this dll when JIT compilation starts. The NVRTC location has been added to the search path after the call
load_nvidia_dynamic_lib('nvrtc'), as described in the linked comment.@ZzEeKkAa could you provide a better reproducer to the true failure?
rwgk commentedon Aug 18, 2025
@leofang, @ZzEeKkAa and I discussed this offline, we agree.
leofang commentedon Aug 18, 2025
Thanks guys, let's close this for now.