Skip to content

CI: CUDA version mismatch in 13.0.2 local test jobs #1691

Closed

Description

@cpcloud

Summary

The 13.0.2, local CI test jobs install nvidia-cuda-runtime-13.1.80 via pip (pulled in by cuda-toolkit[cudart]==13.*) alongside a local CTK 13.0.2. This creates a version conflict: the local nvjitlink is 13.0.88 but libcudadevrt.a from the pip wheel is 13.1.80.

How it was discovered

PR #1690 updates test_device_launch.py to use cuda.pathfinder.find_static_lib("cudadevrt") instead of a manual CUDA_HOME-only lookup. The pathfinder searches site-packages first (by design, matching all other pathfinder finders), so it returns the 13.1 libcudadevrt.a from the pip wheel. When nvjitlink 13.0 tries to link it:

cuda.bindings.nvjitlink.nvJitLinkError: ERROR_INVALID_INPUT (3)
Linker error log: ERROR 4: bad input:/opt/hostedtoolcache/Python/3.13.12/x64/lib/python3.13/site-packages/nvidia/cu13/lib/libcudadevrt.a

The old test code avoided this by only looking at CUDA_HOME, which pointed to the local 13.0.2 CTK.

Affected jobs

  • py3.13, 13.0.2, local, h100
  • py3.13, 13.0.2, local, rtxpro6000
  • py3.10, 13.0.2, local, rtxpro6000 (TCC) (Windows)

Why other jobs pass

  • 13.1.1, local jobs: the local CTK and pip wheel are both 13.1.x, so no mismatch.
  • 13.0.2, wheels jobs: the device launch test is skipped due to architecture (sm_89 < sm_90).
  • 12.9.1, local jobs: no cu13 pip wheel conflict.

Activity

  1. added a commit that references this issue on Feb 26, 2026
    c2a7667
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triageNeeds the team's attention

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions