[BUG]: pathfinder load_with_system_search tries to load old libraries first

This is more-or-less a sub-issue of #654

Is this a duplicate?

I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Silent Failure

Component

cuda.pathfinder

Describe the bug

I've been trying to work out why, when I have cuda-nvrtc 13.0.48 in my conda environment, pathfinder seems to always pick an old system-installed libnvrtc. First it would find one from the toolkit in /usr/local/cuda-11.2, then I deleted that and it picked one up from /usr/local/cuda-12.0.

I think the issue is that the list of NVRTC sonames to search looks like:

    "nvrtc": (
        "libnvrtc.so.11.0",
        "libnvrtc.so.11.1",
        "libnvrtc.so.11.2",
        "libnvrtc.so.12",
        "libnvrtc.so.13",
    ),

and it will try to dlopen() each of these via ctypes.CDLL until it succeeds, then use whatever it found:

cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_dl_linux.py

Lines 101 to 109 in b87c787

    
           for soname in candidate_sonames: 
        
               try: 
        
                   handle = ctypes.CDLL(soname, CDLL_MODE) 
        
                   abs_path = abs_path_for_dynamic_library(libname, handle) 
        
                   if abs_path is None: 
        
                       raise RuntimeError(f"No expected symbol for {libname=!r}") 
        
                   return LoadedDL(abs_path, False, handle._handle) 
        
               except OSError: 
        
                   pass

How to Reproduce

I think these steps provide a minimal reproducer:

Install an old CUDA toolkit (e.g. 12.0) in the usual system location, and presumably as a result of that it will show up in e.g. ldconfig -p.

Create an environment with a newer NVRTC, e.g.

mamba create -n test-pathfinder python=3.13 cuda-python cuda-core cuda-nvrtc cuda-version=13

Run the following:

from cuda import pathfinder

print(pathfinder.load_nvidia_dynamic_lib("nvrtc"))

This will give

LoadedDL(abs_path='/usr/local/cuda-12/targets/x86_64-linux/lib/libnvrtc.so.12', was_already_loaded_from_elsewhere=False, _handle_uint=94470242743920)

I verified that the behaviour occurs because of the conditions outlined above by moving "libnvrtc.so.13" to the top of the list of sonames, which then makes the reproducer output

LoadedDL(abs_path='/home/gmarkall/miniforge3/envs/numbadev/lib/python3.12/lib-dynload/../../libnvrtc.so.13', was_already_loaded_from_elsewhere=False, _handle_uint=95396331422112)

Expected behavior

That nvrtc is loaded from the conda environment instead, giving e.g.

LoadedDL(abs_path='/home/gmarkall/miniforge3/envs/numbadev/lib/python3.12/lib-dynload/../../libnvrtc.so.13', was_already_loaded_from_elsewhere=False, _handle_uint=95396331422112)

Operating System

Linux

nvidia-smi output

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.65.06              Driver Version: 580.65.06      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX 3500 Ada Gene...    On  |   00000000:01:00.0 Off |                  Off |
| N/A   44C    P3            593W /   71W |      15MiB /  12282MiB |     17%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            3035      G   /usr/lib/xorg/Xorg                        4MiB |
+-----------------------------------------------------------------------------------------+

@gmarkall

Sorry for the trouble @gmarkall, this was already high up on my mind.

I was itching several times to change sorted(...) to reversed(sorted(...)) in the loops over the tabulated .so files, but always backed off because we want to inspect the driver version before deciding which of the tabulated files are compatible:

#654

Without something like that, it can go wrong either way.

It looks like I underestimated by a lot how often people would have mixed environments.

We (mainly @kkraus14 and @leofang) need to decide if I should jump on this now or later.

@rwgk

@rwgk I think the issue unfortunately goes beyond the driver version, i.e. with the 580 driver 13, 12, and 11 will all be compatible where in Graham's case say he had a CUDA 12 conda environment and a CUDA 13 system install, it would still erroneously find the CUDA 13 libs on the system.

Some ideas on best path forward here:

We could allow a user to pass a major CUDA version to the load_nvidia_dynamic_lib call which we can then use to determine the versioned filename to look for. We could keep API compatibility by defaulting it to and supporting None in which case we maintain searching across multiple versions (probably should do newest to oldest).
We could special case searching in the conda environment similar to what we do for wheels and handle that before we do the general system search via CDLL (dlopen). Should still probably reverse the order and do newest to oldest instead though.

I lean towards doing the latter because there's many situations where you don't know the toolkit version that is installed and so you need to do a search and you always risk a conda environment mismatch if you do a general dlopen based search.

@kkraus14

say he had a CUDA 12 conda environment and a CUDA 13 system install, it would still erroneously find the CUDA 13 libs on the system.

@kkraus14 I don't think this is true. Conda relies on RPATH so it always has the highest precedence. ld.so.cache (which system packages depend on) has a lower precedence.

We could allow a user to pass a major CUDA version to the load_nvidia_dynamic_lib call which we can then use to determine the versioned filename to look for. We could keep API compatibility by defaulting it to and supporting None in which case we maintain searching across multiple versions (probably should do newest to oldest).

I think regardless this is a good feature that we should add, to allow more flexibility without breaking existing code.

We could special case searching in the conda environment

As mentioned, this should not be needed.

Should still probably reverse the order and do newest to oldest instead though.

Agreed. We definitely should search from the latest to the oldest.

@kkraus14

@kkraus14 I don't think this is true. Conda relies on RPATH so it always has the highest precedence. ld.so.cache (which system packages depend on) has a lower precedence.

The precedence doesn't matter if pathfinder is searching for libnvrtc.so.13 and my environment contains libnvrtc.so.12 - it will find the system-installed libnvrtc.so.13 and not bother to search for libnvrtc.so.12, which is in the conda environment (assuming we reversed the search order from newest to oldest - I was having the problem with oldest-to-newest and it finding libnvrtc.so.11.2 though).

We could special case searching in the conda environment

As mentioned, this should not be needed.

I don't want numba-cuda to have to determine what CUDA toolkit version is installed in the conda environment (or if there even is a conda environment) to be able to tell cuda.pathfinder the right version to search for. How could special-casing be avoided without that?

@leofang

@leofang the current logic is that we search for each supported SONAME, in order for pip wheels, then try each supported SONAME via dlopen (via ctypes), and then try each supported SONAME in $CUDA_HOME. We search for these in the order we have them in the tuple we store i.e. ('libnvrtc.so.11', 'libnvrtc.so.12', 'libnvrtc.so.13').

So the overall search order looks like (assuming it's not loaded in process already):

pip wheel libnvrtc.so.11
pip wheel libnvrtc.so.12
pip wheel libnvrtc.so.13
dlopen libnvrtc.so.11
1. RPATH libnvrtc.so.11 (conda)
2. $LD_LIBRARY_PATH libnvrtc.so.11
3. RUNPATH libnvrtc.so.11
4. ld.so.cache libnvrtc.so.11
5. /lib libnvrtc.so.11
6. /usr/lib libnvrtc.so.11
dlopen libnvrtc.so.12
1. RPATH libnvrtc.so.12 (conda)
2. $LD_LIBRARY_PATH libnvrtc.so.12
3. RUNPATH libnvrtc.so.12
4. ld.so.cache libnvrtc.so.12
5. /lib libnvrtc.so.12
6. /usr/lib libnvrtc.so.12
dlopen libnvrtc.so.13
1. RPATH libnvrtc.so.13 (conda)
2. $LD_LIBRARY_PATH libnvrtc.so.13
3. RUNPATH libnvrtc.so.13
4. ld.so.cache libnvrtc.so.13
5. /lib libnvrtc.so.13
6. /usr/lib libnvrtc.so.13
$CUDA_HOME libnvrtc.so.11
$CUDA_HOME libnvrtc.so.12
$CUDA_HOME libnvrtc.so.13

Even if we reversed the order to 13 first, you run into the issue where if you have a CUDA 12 conda environment, you'll end up finding a system installed library due to the way we use dlopen to handle the conda environment not giving us an opportunity to try all possible versions of a library before falling back to other system locations.

Discussed in the Aug 26 meeting. We decided to make this change (reverting search order), which is a behavioral-change that warrants a minor release (v1.2.0) instead of a patch release (v1.1.1). This is also requested by the nvmath-python PIC.

The other argument in favor of this change is: there should be a separation of concern, to keep our project scope well-defined.
Ideally this should be the package manager's job to resolve into a sane, consistent environment, in which every installed CUDA component comes from the same CUDA major.minor version. There are other initiatives such as WheelNext that would offer a more robust solution in the future. The pathfinder should just be responsible for finding/loading by design, assuming the right packages are already properly installed.

Finally, in any case this does not hold,

say he had a CUDA 12 conda environment and a CUDA 13 system install, it would still erroneously find the CUDA 13 libs on the system.

Even if we reversed the order to 13 first, you run into the issue where if you have a CUDA 12 conda environment, you'll end up finding a system installed library due to the way we use dlopen to handle the conda environment not giving us an opportunity to try all possible versions of a library before falling back to other system locations.

I confirmed my understanding (that conda will always be honored and CUDA 12 will be loaded) by running the pathfinder v1.1.0 with a conda env of CUDA 12 inside a CUDA 13 container (all outputs omitted except for the last step):

$ docker run --rm -it --gpus=all nvidia/cuda:13.0.0-devel-ubuntu22.04  # drop into the container
$ ldconfig -p | grep nvrtc  # check nvrtc exists in search path
$ apt update && apt install -y wget vim
$ wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
$ bash Miniforge3-Linux-x86_64.sh
$ source /root/miniforge3/bin/activate
$ conda create -n test2 -y python=3.13 cuda-version=12 cuda-bindings 
$ conda activate test2   
$ python -c "from cuda.pathfinder import load_nvidia_dynamic_lib; print(load_nvidia_dynamic_lib('nvrtc'))"                                     
LoadedDL(abs_path='/root/miniforge3/envs/test2/lib/python3.13/lib-dynload/../../libnvrtc.so.12', was_already_loaded_from_elsewhere=False, _handle_uint=94317382465616)

I confirmed my understanding (that conda will always be honored and CUDA 12 will be loaded) by running the pathfinder v1.1.0 with a conda env of CUDA 12 inside a CUDA 13 container (all outputs omitted except for the last step):

Please try the same with a CUDA 11 container. If there's a system CUDA 11 install it will find that first erroneously.

I believe your example above only works because we prioritize 11 then 12 then 13 currently. This is because in SUPPORTED_LINUX_SONAMES_CTK we order it that way, i.e. for nvrtc:

cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/supported_nvidia_libs.py

Lines 227 to 233 in 1137e15

    
           "nvrtc": ( 
        
               "libnvrtc.so.11.0", 
        
               "libnvrtc.so.11.1", 
        
               "libnvrtc.so.11.2", 
        
               "libnvrtc.so.12", 
        
               "libnvrtc.so.13", 
        
           ),

At search time we eventually call

cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_dl_linux.py

Line 151 in 1137e15

def load_with_system_search(libname: str) -> Optional[LoadedDL]:

which uses

cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_dl_linux.py

Lines 127 to 130 in 1137e15

    
           def get_candidate_sonames(libname: str) -> list[str]: 
        
               candidate_sonames = list(SUPPORTED_LINUX_SONAMES.get(libname, ())) 
        
               candidate_sonames.append(f"lib{libname}.so") 
        
               return candidate_sonames

which uses that oldest to newest list without reordering them.

But we also discussed in the meeting, right? That usually conda envs have a newer CUDA installed compared to the system one, if any. By reversing the order, we make this common case served. There is no simple way to make both directions happy.

And yes, using CUDA 11 container + CUDA 12 conda + public pathfinder v1.1.0 (so the search order is not yet reverted) we get CUDA 11 NVRTC loaded

(test2) root@f4662ad7a6c0:/# python -c "from cuda.pathfinder import load_nvidia_dynamic_lib; print(load_nvidia_dynamic_lib('nvrtc'))"                                      
LoadedDL(abs_path='/usr/local/cuda/targets/x86_64-linux/lib/libnvrtc.so.11.2', was_already_loaded_from_elsewhere=False, _handle_uint=93907934923248)

But we also discussed in the meeting, right? That usually conda envs have a newer CUDA installed compared to the system one, if any. By reversing the order, we make this common case served. There is no simple way to make both directions happy.

Yes, the more common case is having a newer conda env than the system install, but the reverse still occurs as well where you need to support an older env with a newer system install. Reverting the search order is better with the current search behavior, but I still think it's going to cause too many issues for users if we don't address it.

The "simple" way to make both directions happy is to detect being in a conda environment (or in conda build) and searching for the supported SONAMES within it before doing the general system search.

Navigation Menu

Repository navigation

[BUG]: pathfinder `load_with_system_search` tries to load old libraries first #839

Is this a duplicate?

Type of Bug

Component

Describe the bug

How to Reproduce

Expected behavior

Operating System

nvidia-smi output

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	for soname in candidate_sonames:
	try:
	handle = ctypes.CDLL(soname, CDLL_MODE)
	abs_path = abs_path_for_dynamic_library(libname, handle)
	if abs_path is None:
	raise RuntimeError(f"No expected symbol for {libname=!r}")
	return LoadedDL(abs_path, False, handle._handle)
	except OSError:
	pass

	"nvrtc": (
	"libnvrtc.so.11.0",
	"libnvrtc.so.11.1",
	"libnvrtc.so.11.2",
	"libnvrtc.so.12",
	"libnvrtc.so.13",
	),

	def get_candidate_sonames(libname: str) -> list[str]:
	candidate_sonames = list(SUPPORTED_LINUX_SONAMES.get(libname, ()))
	candidate_sonames.append(f"lib{libname}.so")
	return candidate_sonames

[BUG]: pathfinder load_with_system_search tries to load old libraries first #839

Description

Is this a duplicate?

Type of Bug

Component

Describe the bug

How to Reproduce

Expected behavior

Operating System

nvidia-smi output

Activity

rwgk commented on Aug 15, 2025

kkraus14 commented on Aug 15, 2025

kkraus14 commented on Aug 15, 2025

leofang commented on Aug 19, 2025

gmarkall commented on Aug 19, 2025

kkraus14 commented on Aug 19, 2025

leofang commented on Aug 28, 2025

kkraus14 commented on Aug 28, 2025

leofang commented on Aug 28, 2025

leofang commented on Aug 28, 2025

kkraus14 commented on Aug 28, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

[BUG]: pathfinder `load_with_system_search` tries to load old libraries first #839