You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CI: CUDA version mismatch in 13.0.2 local test jobs #1691
The 13.0.2, local CI test jobs install nvidia-cuda-runtime-13.1.80 via pip (pulled in by cuda-toolkit[cudart]==13.*) alongside a local CTK 13.0.2. This creates a version conflict: the local nvjitlink is 13.0.88 but libcudadevrt.a from the pip wheel is 13.1.80.
How it was discovered
PR #1690 updates test_device_launch.py to use cuda.pathfinder.find_static_lib("cudadevrt") instead of a manual CUDA_HOME-only lookup. The pathfinder searches site-packages first (by design, matching all other pathfinder finders), so it returns the 13.1 libcudadevrt.a from the pip wheel. When nvjitlink 13.0 tries to link it:
cuda.bindings.nvjitlink.nvJitLinkError: ERROR_INVALID_INPUT (3)
Linker error log: ERROR 4: bad input:/opt/hostedtoolcache/Python/3.13.12/x64/lib/python3.13/site-packages/nvidia/cu13/lib/libcudadevrt.a
The old test code avoided this by only looking at CUDA_HOME, which pointed to the local 13.0.2 CTK.
Affected jobs
py3.13, 13.0.2, local, h100
py3.13, 13.0.2, local, rtxpro6000
py3.10, 13.0.2, local, rtxpro6000 (TCC) (Windows)
Why other jobs pass
13.1.1, local jobs: the local CTK and pip wheel are both 13.1.x, so no mismatch.
13.0.2, wheels jobs: the device launch test is skipped due to architecture (sm_89 < sm_90).
Activity
feat(pathfinder): support finding static libraries like libcudadevrt.a (