[BUG]: KeyError in load_dl_linux.py:47 EXPECTED_LIB_SYMBOLS[libname]

Is this a duplicate?

I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug

Runtime Error

Component

cuda.pathfinder

Describe the bug

Under certain circumstances code responsible for loading ctk libraries triggers key error for a library that is not part of ctk

How to Reproduce

python -m venv .venv
source .venv/bin/activate
pip install cuda-pathfinder
export CUDA_HOME=/usr/local/cuda12.9
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
wget -q https://developer.download.nvidia.com/compute/cudss/redist/libcudss/linux-x86_64/libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
tar -xvf libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
export LD_LIBRARY_PATH=$(pwd)/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:$LD_LIBRARY_PATH
python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('cudss'))"

Error:

python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('cudss'))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 71, in load_nvidia_dynamic_lib
    return _load_lib_no_cache(libname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_nvidia_dynamic_lib.py", line 40, in _load_lib_no_cache
    loaded = load_with_system_search(libname)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 104, in load_with_system_search
    abs_path = abs_path_for_dynamic_library(libname, handle)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/scratch.yhavrylko_ent/Projects/nvidia/nvmath-python/.venv-sysctk12/lib/python3.11/site-packages/cuda/pathfinder/_dynamic_libs/load_dl_linux.py", line 47, in abs_path_for_dynamic_library
    for symbol_name in EXPECTED_LIB_SYMBOLS[libname]:
                       ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
KeyError: 'cudss'

Expected behavior

Find library with respect to LD_LIBRARY_PATH or at least return proper runtime error that library was not found (not preferable, since with nvidia-cudss-cu12 it loads library properly).

Operating System

No response

nvidia-smi output

No response

since with nvidia-cudss-cu12 it loads library properly).

What does this mean?

Find library with respect to LD_LIBRARY_PATH or at least return proper runtime error that library was not found (not preferable, since with nvidia-cudss-cu12 it loads library properly).

The library was actually found and loaded.

To figure out the absolute path under Linux, we need a trick (I researched this a lot):

cuda-python/cuda_pathfinder/cuda/pathfinder/_dynamic_libs/load_dl_linux.py

Lines 54 to 58 in b87c787

    
           addr = ctypes.cast(symbol, ctypes.c_void_p) 
        
           info = DlInfo() 
        
           if LIBDL.dladdr(addr, ctypes.byref(info)) == 0: 
        
               raise OSError(f"dladdr failed for {libname=!r}") 
        
           return info.dli_fname.decode()  # type: ignore[no-any-return]

We need to get our hands on any one symbol in the library, and use that as a wedge (so to speak) to retrieve info.dli_fname.

This is really quite silly, but I haven't found a better way.

So, to support cudss, really all we need is one symbol name.

I'll give it a quick try.

since with nvidia-cudss-cu12 it loads library properly).

What does this mean?

The abs_path_for_dynamic_library() function is called only from check_if_already_loaded_from_elsewhere() and load_with_system_search(), but not if we figured out the absolute path already by searching site-packages.

Quick update: ChatGPT 5 suggests we don't need EXPECTED_LIB_SYMBOLS. If (!) that's true, it's a game changer, so I'll try it out now.

PR #834 fixes this issue (in a very general way):

Local testing:

#!/bin/bash
set -e -x
pwd
git switch linux_abs_path_from_handle_v2
git log -n 1
git clean -fdx
python3 -m venv Cp12WslVenv
. Cp12WslVenv/bin/activate
pip install --upgrade pip
cd cuda_pathfinder/
pip install .
wget -q https://developer.download.nvidia.com/compute/cudss/redist/libcudss/linux-x86_64/libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
tar -xvf libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
export LD_LIBRARY_PATH=$(pwd)/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:$LD_LIBRARY_PATH
python -c "from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('cudss'))"

Output:

+ pwd
/home/rgrossekunst/forked/cuda-python
+ git switch linux_abs_path_from_handle_v2
Already on 'linux_abs_path_from_handle_v2'
Your branch is up to date with 'origin/linux_abs_path_from_handle_v2'.
+ git log -n 1
commit e716f1cb2f2a5039a7d46d341aa09b9072ac8ff7
Author: Ralf W. Grosse-Kunstleve <[email protected]>
Date:   Tue Aug 12 22:54:07 2025 -0700

    load_dl_windows.py: avoid unnecessary function-level imports (this was just an oversight)
+ git clean -fdx
Removing Cp12WslVenv/
Removing cuda_pathfinder/cuda/pathfinder/__pycache__/
Removing cuda_pathfinder/cuda/pathfinder/_dynamic_libs/__pycache__/
Removing cuda_pathfinder/cuda/pathfinder/_utils/__pycache__/
Removing cuda_pathfinder/cuda_pathfinder.egg-info/
+ python3 -m venv Cp12WslVenv
+ . Cp12WslVenv/bin/activate
++ deactivate nondestructive
++ '[' -n '' ']'
++ '[' -n '' ']'
++ hash -r
++ '[' -n '' ']'
++ unset VIRTUAL_ENV
++ unset VIRTUAL_ENV_PROMPT
++ '[' '!' nondestructive = nondestructive ']'
++ '[' linux-gnu = cygwin ']'
++ '[' linux-gnu = msys ']'
++ export VIRTUAL_ENV=/home/rgrossekunst/forked/cuda-python/Cp12WslVenv
++ VIRTUAL_ENV=/home/rgrossekunst/forked/cuda-python/Cp12WslVenv
++ _OLD_VIRTUAL_PATH=/home/rgrossekunst/forked/cuda-python/Cp12WslVenv/bin:/home/rgrossekunst/rwgk_config_nvidia/bin:.:/home/rgrossekunst/rwgk_config/bin:/home/rgrossekunst/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/snap/bin
++ PATH=/home/rgrossekunst/forked/cuda-python/Cp12WslVenv/bin:/home/rgrossekunst/forked/cuda-python/Cp12WslVenv/bin:/home/rgrossekunst/rwgk_config_nvidia/bin:.:/home/rgrossekunst/rwgk_config/bin:/home/rgrossekunst/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/snap/bin
++ export PATH
++ '[' -n '' ']'
++ '[' -z '' ']'
++ _OLD_VIRTUAL_PS1=
++ PS1='(Cp12WslVenv) '
++ export PS1
++ VIRTUAL_ENV_PROMPT='(Cp12WslVenv) '
++ export VIRTUAL_ENV_PROMPT
++ hash -r
+ pip install --upgrade pip
Requirement already satisfied: pip in ./Cp12WslVenv/lib/python3.12/site-packages (24.0)
Collecting pip
  Using cached pip-25.2-py3-none-any.whl.metadata (4.7 kB)
Using cached pip-25.2-py3-none-any.whl (1.8 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 24.0
    Uninstalling pip-24.0:
      Successfully uninstalled pip-24.0
Successfully installed pip-25.2
+ cd cuda_pathfinder/
+ pip install .
Processing /home/rgrossekunst/forked/cuda-python/cuda_pathfinder
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
Building wheels for collected packages: cuda-pathfinder
  Building wheel for cuda-pathfinder (pyproject.toml): started
  Building wheel for cuda-pathfinder (pyproject.toml): finished with status 'done'
  Created wheel for cuda-pathfinder: filename=cuda_pathfinder-1.1.1a0-py3-none-any.whl size=17297 sha256=307a7fa1ec404d204e363864a75768a174a028caaac590a73177008c9e766a40
  Stored in directory: /tmp/pip-ephem-wheel-cache-lekk4y1h/wheels/d1/04/8c/c856bb3f85f3511d35e8d69acc9ca63f99f4d52d879aba968a
Successfully built cuda-pathfinder
Installing collected packages: cuda-pathfinder
Successfully installed cuda-pathfinder-1.1.1a0
+ wget -q https://developer.download.nvidia.com/compute/cudss/redist/libcudss/linux-x86_64/libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
+ tar -xvf libcudss-linux-x86_64-0.5.0.16_cuda12-archive.tar.xz
libcudss-linux-x86_64-0.5.0.16_cuda12-archive/
... <snip> ...
libcudss-linux-x86_64-0.5.0.16_cuda12-archive/LICENSE
++ pwd
+ export LD_LIBRARY_PATH=/home/rgrossekunst/forked/cuda-python/cuda_pathfinder/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:/home/rgrossekunst/z/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:
+ LD_LIBRARY_PATH=/home/rgrossekunst/forked/cuda-python/cuda_pathfinder/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:/home/rgrossekunst/z/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib:
+ python -c 'from cuda import pathfinder as pf; print(pf.load_nvidia_dynamic_lib('\''cudss'\''))'
LoadedDL(abs_path='/home/rgrossekunst/forked/cuda-python/cuda_pathfinder/libcudss-linux-x86_64-0.5.0.16_cuda12-archive/lib/libcudss.so', was_already_loaded_from_elsewhere=False, _handle_uint=21876640)

since with nvidia-cudss-cu12 it loads library properly).

What does this mean?

If I install cudss as a wheel - library is being loaded through the pathfinder correctly. If I install cudss as a tarball and set LD_LIBRARY_PATH, it runs into exception.

Navigation Menu

Repository navigation

[BUG]: KeyError in load_dl_linux.py:47 EXPECTED_LIB_SYMBOLS[libname] #833

Is this a duplicate?

Type of Bug

Component

Describe the bug

How to Reproduce

Expected behavior

Operating System

nvidia-smi output

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	addr = ctypes.cast(symbol, ctypes.c_void_p)
	info = DlInfo()
	if LIBDL.dladdr(addr, ctypes.byref(info)) == 0:
	raise OSError(f"dladdr failed for {libname=!r}")
	return info.dli_fname.decode() # type: ignore[no-any-return]

[BUG]: KeyError in load_dl_linux.py:47 EXPECTED_LIB_SYMBOLS[libname] #833

Description

Is this a duplicate?

Type of Bug

Component

Describe the bug

How to Reproduce

Expected behavior

Operating System

nvidia-smi output

Activity

leofang commented on Aug 13, 2025

rwgk commented on Aug 13, 2025

rwgk commented on Aug 13, 2025

rwgk commented on Aug 13, 2025

ZzEeKkAa commented on Aug 13, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions