[FEA]: Support finding nvvm from the system

Is this a duplicate?

I confirmed there appear to be no duplicate issues for this request and that I agree to the Code of Conduct

Area

cuda.pathfinder

Is your feature request related to a problem? Please describe.

numba-cuda currently supports functioning with only system packages and no nvvm/nvrtc wheels. Today, I can't find nvvm from the system using cuda-pathfinder:

root@machine:/# ls /usr/local/cuda/nvvm/lib64/libnvvm.so
/usr/local/cuda/nvvm/lib64/libnvvm.so

>>> from cuda.pathfinder import load_nvidia_dynamic_lib
>>> load_nvidia_dynamic_lib('nvvm')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/pyenv/versions/3.12.11/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 140, in load_nvidia_dynamic_lib
    return _load_lib_no_cache(libname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.12.11/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 57, in _load_lib_no_cache
    finder.raise_not_found_error()
  File "/pyenv/versions/3.12.11/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/find_nvidia_dynamic_lib.py", line 210, in raise_not_found_error
    raise DynamicLibNotFoundError(f'Failure finding "{self.lib_searched_for}": {err}\n{att}')
cuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError: Failure finding "libnvvm.so": No such file: libnvvm.so*, No such file: libnvvm.so*, No such file: libnvvm.so*, No such file: libnvvm.so*

Describe the solution you'd like

I'd like load_nvidia_dynamic_lib to be able to load the library at the above path.

Describe alternatives you've considered

numba-cuda currently implements this handling so we could theoretically maintain it, but then it somewhat defeats the purpose of adopting pathfinder.

Additional context

No response

With which version of pathfinder does this happen? I thought we've made a special case for nvvm precisely because it does not show up in the system search path. We also covered this case in the CI ($CUDA_PATH/nvvm/lib64/ is NOT added to $LD_LIBRARY_PATH).

The error is present for 1.3.1. The issue may be reproduced inside

rapidsai/citestwheel:cuda12.9.1-ubuntu24.04-py3.12

with

pip install cuda-pathfinder==1.3.1

Then

root@eb59577e416b:/# find . -name "libnvvm.so"
./usr/local/cuda-12.9/nvvm/lib64/libnvvm.so
root@eb59577e416b:/# python
Python 3.12.11 (main, Sep 22 2025, 15:22:20) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from cuda.pathfinder import load_nvidia_dynamic_lib
>>> load_nvidia_dynamic_lib('nvvm')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/pyenv/versions/3.12.11/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 140, in load_nvidia_dynamic_lib
    return _load_lib_no_cache(libname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pyenv/versions/3.12.11/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 57, in _load_lib_no_cache
    finder.raise_not_found_error()
  File "/pyenv/versions/3.12.11/lib/python3.12/site-packages/cuda/pathfinder/_dynamic_libs/find_nvidia_dynamic_lib.py", line 210, in raise_not_found_error
    raise DynamicLibNotFoundError(f'Failure finding "{self.lib_searched_for}": {err}\n{att}')
cuda.pathfinder._dynamic_libs.load_dl_common.DynamicLibNotFoundError: Failure finding "libnvvm.so": No such file: libnvvm.so*, No such file: libnvvm.so*, No such file: libnvvm.so*, No such file: libnvvm.so*

I thought we've made a special case for nvvm precisely because it does not show up in the system search path.

In the meantime (between #441 and #1038) it became certain to me that we need to tie the discovery of nvvm in some way to the discovery another library, at the point where we reach the system-search stage, but nvvm is not found there. — That's actually what led me to propose the mechanism under #1038, although that mechanism is powerful beyond just finding nvvm in a consistent way.

As soon as I'm done with the CTK-next work, I'll jump on this. I'll probably solve this issue with a "scoped search lite" approach.

This seems like another case where libraries are in /usr/local/cuda* but are not necessarily in the ldconfig and that the expectation is we still search and find them within there regardless

@kkraus14

@kkraus14 wrote:

This seems like another case where libraries are in /usr/local/cuda* but are not necessarily in the ldconfig and that the expectation is we still search and find them within there regardless

Yes. I was thinking libvvm is the only such case, by design.

It's a long-established numba-cuda feature that libnvvm is found, unless we are OK with a regression from a user perspective, we have to support it.

Tangential, but maybe useful for me to know long-term:

I'm only aware of one other such case, but we're considering that a bug:

cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_dl_linux.py

Lines 179 to 195 in 53f5680

    
           def _work_around_known_bugs(libname: str, found_path: str) -> None: 
        
               if libname == "nvrtc": 
        
                   # Work around bug/oversight in 
        
                   #   nvidia_cuda_nvrtc-13.0.48-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl 
        
                   # Issue: libnvrtc.so.13 RUNPATH is not set. 
        
                   # This workaround is highly specific 
        
                   #   - for simplicity. 
        
                   #   - to not mask bugs in future nvidia-cuda-nvrtc releases. 
        
                   #   - because a more general workaround is complicated. 
        
                   dirname, basename = os.path.split(found_path) 
        
                   if basename == "libnvrtc.so.13": 
        
                       dep_basename = "libnvrtc-builtins.so.13.0" 
        
                       dep_path = os.path.join(dirname, dep_basename) 
        
                       if os.path.isfile(dep_path): 
        
                           # In case of failure, defer to primary load, which is almost certain to fail, too. 
        
                           with contextlib.suppress(OSError): 
        
                               ctypes.CDLL(dep_path, CDLL_MODE)

Are there any other cases?

I'm only aware of one other such case, but we're considering that a bug:

I just looked around, I think the bug that prompted us to add the _work_around_known_bugs() was fixed already.

I ran into a similar problem elsewhere (internal) and wrongly assumed it's still the same bug.

I still need to understand why exactly I was seeing that similar problem.

This seems like another case where libraries are in /usr/local/cuda* but are not necessarily in the ldconfig and that the expectation is we still search and find them within there regardless

Yes. I was thinking libvvm is the only such case, by design.

Yes. But, this question still needs to be answered:

We also covered this case in the CI ($CUDA_PATH/nvvm/lib64/ is NOT added to $LD_LIBRARY_PATH).

Why didn't we run into any issue so far, in our local-ctk CI pipelines? My impression is that we already addressed this. Would be nice to understand it before implementing any fix. Maybe there is nothing to fix?

Why didn't we run into any issue so far, in our local-ctk CI pipelines?

They all set CUDA_HOME, libnvvm is found that way.

They all set CUDA_HOME, libnvvm is found that way.

I see, this is what I missed, thanks!

This is now supported and in use in numba-cuda. Closing, thanks all!

Navigation Menu

Repository navigation

[FEA]: Support finding `nvvm` from the system #1157

Is this a duplicate?

Area

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def _work_around_known_bugs(libname: str, found_path: str) -> None:
	if libname == "nvrtc":
	# Work around bug/oversight in
	# nvidia_cuda_nvrtc-13.0.48-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl
	# Issue: libnvrtc.so.13 RUNPATH is not set.
	# This workaround is highly specific
	# - for simplicity.
	# - to not mask bugs in future nvidia-cuda-nvrtc releases.
	# - because a more general workaround is complicated.
	dirname, basename = os.path.split(found_path)
	if basename == "libnvrtc.so.13":
	dep_basename = "libnvrtc-builtins.so.13.0"
	dep_path = os.path.join(dirname, dep_basename)
	if os.path.isfile(dep_path):
	# In case of failure, defer to primary load, which is almost certain to fail, too.
	with contextlib.suppress(OSError):
	ctypes.CDLL(dep_path, CDLL_MODE)

[FEA]: Support finding nvvm from the system #1157

Description

Is this a duplicate?

Area

Is your feature request related to a problem? Please describe.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

Activity

leofang commented on Oct 21, 2025

brandon-b-miller commented on Oct 21, 2025

rwgk commented on Oct 21, 2025

kkraus14 commented on Oct 27, 2025

rwgk commented on Oct 27, 2025

rwgk commented on Oct 27, 2025

leofang commented on Nov 4, 2025

rwgk commented on Nov 4, 2025

leofang commented on Nov 17, 2025

cpcloud commented on Feb 4, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[FEA]: Support finding `nvvm` from the system #1157