[FEA]: Explore pathfinder "scoped search" feature

Problem to solve

The searches implemented in load_nvidia_dynamic_lib() and find_nvidia_header_directory() are completely independent from each other. For example, load_nvidia_dynamic_lib("nvrtc") might resolve to a site-packages path, while load_nvidia_dynamic_lib("nvJitLink") finds a Conda path, and find_nvidia_header_directory("nvvm") locates headers under /usr/local/cuda. This can lead to surprising inconsistencies and confusing behavior deep in the call stack.

Potential approach (with low maintenance as a goal in mind)

Core idea: Introduce the concept of a pivot library that defines the scope for all subsequent searches.

Here, package system means, for example:

/usr/local/cuda (or another "standard" CTK installation)
Conda (including pixi)
site-packages (wheels, PyPI)

How it would work in practice:

A client code author decides which library is most pivotal and loads it first.
search_scope = cuda.pathfinder.determine_search_scope(pivot_libname="nvrtc") — Loads the pivot library and returns it in a wrapper object.
loaded_dl = search_scope.load_nvidia_dynamic_lib("nvJitLink") — Only succeeds if the target library exists in the same package system, otherwise fails with a helpful error.
hdr_dir = search_scope.find_nvidia_header_directory("nvvm") — Similarly constrained to the same package system.

This design shifts responsibility for consistency to package managers, which already provide version alignment via pinning. We avoid duplicating those mechanisms and instead ensure our tooling surfaces clear, consistent results.

@kkraus14

xref to #1090 (comment)

Comment by @kkraus14 there:

If one library has a binary dependency on another library, I don't think compatibility guarantees are made outside of driver version. I.e. lib2 depends on lib1, the only guarantee we currently have is that it works if both lib2 and lib1 are the same patch version. So from the perspective of test_load_nvidia_dynamic_libs.py we could end up in an environment with different patch versions and someone blows up at load time because we load an incompatible dependency (since the dependencies aren't captured in python package dependencies to support using a system CTK).

Cross-referencing here since this might be something we could guard against (e.g. helpful error message) as part of the scoped search feature.

Tracking prompt and response:

Prompt

I want to approach the work from a practical angle, roughly:
* come up with a minimal viable product
* that'll already force us to think about compatibility details while we work on it
* iterate based on the first version, building up a test suite as we go

To establish relevant context, could you please make a quick pass reading all code under cuda_pathfinder/cuda/pathfinder?

It's not a lot of code:

smc120-0009.ipp2a2.colossus.nvidia.com:/wrk/forked/cuda-python/cuda_pathfinder/cuda/pathfinder $ cloc .
      28 text files.
      28 unique files.
       0 files ignored.

github.com/AlDanial/cloc v 1.98  T=0.03 s (984.6 files/s, 118468.9 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
Python                          27            587            604           2175
Markdown                         1              1              0              2
-------------------------------------------------------------------------------
SUM:                            28            588            604           2177
-------------------------------------------------------------------------------

Between writing up issue #1038 and now, the "CTK root canary probe" feature was introduced into pathfinder, which is game changing: we don't need a pivot library anymore.

We can fully focus on offering "guard rails"

There will be some overlap between our own guard rails, and what package managers provide (most notably pip/uv, Conda). We want to rely on the guard rails provided by package managers within reason, but certainly not blindly.

Let's start with a simple idea and see where we land:

pfchecks = cuda.pathfinder.WithCompatibilityChecks()
pfchecks.load_nvidia_dynamic_lib("nvrtc")
hdr_dir = pfchecks.find_nvidia_header_directory("nvrtc")

Here pfchecks is meant to ensure that the headers are compatible with the dynamic library:
* load_nvidia_dynamic_lib would do the search as already implemented and update the internal state of pfchecks to keep track of what was found
* for find_nvidia_header_directory we have to make a decision:
  * simple alternative: search as already implemented and generate an error if what we find is not compatible with the matching
library
  * more sophisticated alternative: search for a compatible header and report that it was not found, ideally along with a list of
incompatible headers found while searching

The compatibility checks should apply to all combinations of dynamic libs, static libs, headers, and binaries.

The compatibility checks should query the driver version to guard against related incompatibilities

WithCompatibilityChecks() should have optional ctk_major, ctk_minor arguments, with ==, <, <=, >, >= support.

Where applicable (e.g. nvrtc, nvjitlink, nvvm) we may also need to encode that certain version combinations are not supported, even within a given ctk_major line.

How do we approach this systematically?

What we need at the core is a

compatibility_check(driver_version, item1, item2)

function, where, conceptually, we pass any pair of "items" (dynamic lib, static lib, header directory, binary) and it returns if they are compatible, plus a helpful diagnostic if not.

Here driver_version would be what we get from cuDriverGetVersion(), e.g. "12000" or "13020"

I think initially we should limit our focus to packaged_with="ctk", in another iteration we want to add in packaged_with="other" items.

A holistic approach would be to systematically look at all ctk releases we need to cover: 12.0 ... 13.2, but only at the latest patch release for each. The corresponding .run files are:

smc120-0009.ipp2a2.colossus.nvidia.com:/wrk/cuda_linux_run $ ls -l *.run
-r-xr-xr-x 1 rgrossekunst dip 4.0G Jan 28  2023 cuda_12.0.1_525.85.12_linux.run
-r-xr-xr-x 1 rgrossekunst dip 4.1G Apr 17  2023 cuda_12.1.1_530.30.02_linux.run
-r-xr-xr-x 1 rgrossekunst dip 4.1G Aug 23  2023 cuda_12.2.2_535.104.05_linux.run
-r-xr-xr-x 1 rgrossekunst dip 4.1G Dec 12  2023 cuda_12.3.2_545.23.08_linux.run
-r-xr-xr-x 1 rgrossekunst dip 4.2G Mar 28  2024 cuda_12.4.1_550.54.15_linux.run
-r-xr-xr-x 1 rgrossekunst dip 4.1G Jun 25  2024 cuda_12.5.1_555.42.06_linux.run
-r-xr-xr-x 1 rgrossekunst dip 4.2G Nov 14  2024 cuda_12.6.3_560.35.05_linux.run
-r-xr-xr-x 1 rgrossekunst dip 5.1G Mar  3  2025 cuda_12.8.1_570.124.06_linux.run
-r-xr-xr-x 1 rgrossekunst dip 5.5G May 31  2025 cuda_12.9.1_575.57.08_linux.run
-r-xr-xr-x 1 rgrossekunst dip 4.1G Oct  3  2025 cuda_13.0.2_580.95.05_linux.run
-r-xr-xr-x 1 rgrossekunst dip 4.1G Dec 23 08:47 cuda_13.1.1_590.48.01_linux.run
-r-xr-xr-x 1 rgrossekunst dip 4.1G Mar  3 13:48 cuda_13.2.0_595.45.04_linux.run

I installed them already to /usr/local on this workstation:

smc120-0009.ipp2a2.colossus.nvidia.com:/wrk $ ls -ld /usr/local/cuda-*
drwxr-xr-x 17 root root 4096 Apr 10 12:04 /usr/local/cuda-12.0
drwxr-xr-x 17 root root 4096 Apr 10 12:07 /usr/local/cuda-12.1
drwxr-xr-x 17 root root 4096 Apr 10 12:09 /usr/local/cuda-12.2
drwxr-xr-x 17 root root 4096 Apr 10 12:11 /usr/local/cuda-12.3
drwxr-xr-x 17 root root 4096 Apr 10 12:13 /usr/local/cuda-12.4
drwxr-xr-x 17 root root 4096 Apr 10 12:15 /usr/local/cuda-12.5
drwxr-xr-x 17 root root 4096 Apr 10 12:18 /usr/local/cuda-12.6
drwxr-xr-x 17 root root 4096 Apr 10 12:20 /usr/local/cuda-12.8
drwxr-xr-x 17 root root 4096 Apr  5 10:19 /usr/local/cuda-12.9
drwxr-xr-x 17 root root 4096 Apr 10 12:22 /usr/local/cuda-13.0
drwxr-xr-x 16 root root 4096 Jan 16 09:25 /usr/local/cuda-13.1
drwxr-xr-x 16 root root 4096 Mar 15 14:41 /usr/local/cuda-13.2

Based on what you see we support looking at the pathfinder sources, and those /usr/local/cuda-* installations, could you please work out a first implementation of the compatibility_check() function, as independently as possible? I realize item1 and item2 are not well-defined. Could you please ask clarifying questions that we need to answer, to arrive at a minimal viable product?

Response (Cursor 2026.04.08-a41fba1, GPT-5.4 1M Extra High)

First Cut

The cleanest MVP is an internal ResolvedItem abstraction first. Dynamic libs and headers already carry packaged_with in cuda_pathfinder/cuda/pathfinder/_dynamic_libs/descriptor_catalog.py and cuda_pathfinder/cuda/pathfinder/_headers/header_descriptor_catalog.py, but static libs, bitcode libs, and binaries are still separate ad hoc registries in cuda_pathfinder/cuda/pathfinder/_static_libs/find_static_lib.py, cuda_pathfinder/cuda/pathfinder/_static_libs/find_bitcode_lib.py, and cuda_pathfinder/cuda/pathfinder/_binaries/find_nvidia_binary_utility.py.
For CTK-root-based items, version.json is the best source of truth. Every /usr/local/cuda-* install here has one, and it gives both toolkit version and many per-component versions like cuda_nvrtc, libnvjitlink, and cuda_nvcc.
My recommended v1 rule is deliberately conservative: for packaged_with="ctk" items, require the same CTK major.minor; then require driver_major >= ctk_major; treat forward-compat packages as out of scope for now.
compatibility_check() should return a rich result object, not just bool, so a later WithCompatibilityChecks wrapper can raise better diagnostics.

Important Findings

On this workstation, Linux headers live under targets/x86_64-linux/include/..., not just top-level include. Current header fixtures in cuda_pathfinder/tests/test_find_nvidia_headers.py still model top-level include, so the compatibility code should normalize both layouts.
cuda_pathfinder/cuda/pathfinder/_dynamic_libs/search_steps.py already knows how to derive a CTK root from targets/<triple>/lib{,64} paths, so that logic is reusable.
cuda_bindings/cuda/bindings/utils/_version_check.py already uses the simpler policy "warn only if compile major > driver major", which is a good precedent for the first driver guard rail.
nvJitLink.h is especially useful because it encodes the minor line directly in inline symbol names like __nvJitLinkCreate_12_9 and __nvJitLinkCreate_13_0. That makes nvrtc, nvJitLink, and nvvm the best first candidates for stricter same-minor checks later.

MVP Shape

CtkVersion(major, minor)
ResolvedItem(name, kind, packaged_with, abs_path, found_via, ctk_root, ctk_version, component_version)
resolve_*_item(...) helpers for dynamic libs, header dirs, static libs, bitcode libs, and binaries
compatibility_check(driver_version, item1, item2) -> CompatibilityResult

Questions

Is it acceptable that v1 only gives definitive answers for items whose path maps to a CTK root with version.json, and returns insufficient_metadata for the rest?
Do you want v1 to require the same CTK major.minor for any two CTK items, or only the same major with stricter exceptions for nvrtc, nvJitLink, and nvvm?
For driver checks, should v1 use the simple rule driver_major >= ctk_major and explicitly ignore forward-compat packages?
Is it okay that v1 derives static-lib, bitcode-lib, and binary compatibility from the enclosing CTK root version rather than introspecting the artifact itself?
Do you want the first public step to be just the internal compatibility_check() core, with WithCompatibilityChecks added immediately after?

Issue #1038 and PR #1977: where I currently stand (2026-04-28)

High-level summary

Issue 1038 originally started as a broader scoped search idea.
Over time, I split that into two related but distinct directions:
- search scoping / pivot library: once a pivotal library is chosen, later searches stay in the same package system
- compatibility guard rails: detect mixed or incompatible results and fail with helpful errors
While working through PR 1936 and then PR 1977, I made a deliberate decision to focus fully on the second part: catching incompatibilities.
I intentionally did not try to implement full search narrowing / same-origin scoping in PR 1977, because that would make the search logic much more complex and harder to maintain.
My current view is:
- package managers (pip, Conda, etc.) should be responsible for constructing internally consistent environments
- pathfinder should assume that the environment is meant to be internally consistent
- pathfinder should then catch inconsistencies as much as possible and fail with clear diagnostics

How I currently decompose issue 1038

I now think of issue 1038 as having started with two interrelated conceptual parts.

1. Pivot library / scoped search

The original issue text framed this as choosing a pivot library that defines the package-system scope for later searches.

For example:

if the pivot resolves from site-packages, later searches should stay in site-packages
if the pivot resolves from Conda, later searches should stay in Conda
if the pivot resolves under /usr/local/cuda, later searches should stay under that CTK installation

This was the original "scoped search" idea.

2. Compatibility aspects

I think the compatibility side itself has two sub-parts:

narrowing down subsequent searches based on what was already found
catching incompatibilities when results do not line up

PR 1977 addresses the second sub-part, but not the first one.

Where PR #1595 fits in

In the meantime, PR 1595 added the CTK root canary probe feature.

This has a large overlap with the original pivot-library idea, because it gives us a way to infer a CTK root from a system-loaded cudart and then use that information during later searches.

But it is not a full implementation of the original pivot-library / scoped-search concept. It is best understood as:

a useful source of CTK-root information
a partial overlap with the original scoped-search idea
not a full "all later searches are now constrained to this same package system/root" mechanism

Why I decided to focus on catching incompatibilities

While working on PR #1936 / PR 1977, I decided to focus entirely on catching incompatibilities and to defer any attempt at full scoped search.

This was informed by conclusions that came up repeatedly in team discussions:

package managers should be responsible for installing internally consistent environments
pathfinder should not try to re-implement package-manager consistency logic
pathfinder should instead detect inconsistencies as much as possible and fail with helpful error messages

The main practical reason for not implementing search narrowing in PR 1977 is that it would make the search logic much more complex:

each search API would need to understand and preserve package-system scope
more cross-product combinations of site-packages / Conda / CUDA_PATH / canary / system installs would need to be modeled explicitly
search behavior would become harder to reason about and harder to maintain

In short:

PR 1977 is not the full original scope of issue 1038
it is instead a focused, useful subset: process-wide compatibility guard rails

Bird's-eye view of the PR 1977 design

The key architectural move in PR 1977 is to insert a layer of indirection:

process_wide_compatibility_guard_rails: CompatibilityGuardRails = CompatibilityGuardRails()

The public pathfinder APIs then route through this process-wide object.

At the moment this covers:

load_nvidia_dynamic_lib()
locate_nvidia_header_directory()
find_nvidia_header_directory()
locate_static_lib()
find_static_lib()
locate_bitcode_lib()
find_bitcode_lib()
find_nvidia_binary_utility()

Conceptually, this object is a process-wide recorder/checker:

it records what has already been loaded or located through those APIs
each time a new item is resolved, it automatically runs compatibility checks against the already-recorded process state

Why a process-wide singleton?

This is largely determined by the realities of dynamic library loading.

Loading a dynamic library changes process state globally and irreversibly. Once a CUDA dynamic library has been loaded into a process, that process is no longer in a clean pre-load state.

The process-wide singleton models that behavior directly:

there is one evolving process-wide compatibility state
the first CTK-backed item effectively anchors that state
later CTK-backed resolutions are checked against what is already true in the process

So this is not mainly an API-style preference. The realities of dynamic library loading more or less force our hand here.

Side note

The CompatibilityGuardRails class itself is not a singleton.

For unusual situations, callers can construct one or more separate CompatibilityGuardRails() objects and manage their states independently.

What is process-wide is the default public pathfinder behavior, not the class abstraction itself.

What the compatibility checks currently do in PR 1977

This is my current understanding of the v1 behavior.

1. Every resolved item is normalized into metadata

Each located or loaded item is normalized into a ResolvedItem carrying information such as:

what kind of item it is (dynamic-lib, header-dir, static-lib, bitcode-lib, binary)
where it was found
whether it is considered packaged with the CTK, the driver, or "other"
what CTK major.minor line it appears to belong to, if that can be determined

2. CTK metadata is derived in two main ways

For local / conda / system CTK-backed artifacts, PR 1977 primarily derives CTK major.minor from cuda.h.

For wheel-backed artifacts, it can also infer the CTK line from installed cuda-toolkit requirement metadata.

This is an important point: the checks are not just comparing paths heuristically. They rely on explicit metadata sources.

3. Driver version is queried separately

The process-wide guard rails query the CUDA driver version via cuDriverGetVersion() when needed.

That queried driver version becomes part of the compatibility decision-making.

4. The first CTK-backed item becomes the anchor

The first resolved item that is clearly CTK-backed becomes the effective anchor for later comparisons.

That means, for example:

if the first anchored item is nvrtc from CTK 13.2
then a later header directory resolving to CTK 13.0
is rejected as incompatible

This is exactly the behavior demonstrated here:

example of the guard rails in action

5. v1 currently requires an exact CTK major.minor match

The current v1 rule is intentionally conservative:

if two CTK-backed items resolve to different CTK major.minor lines, they are rejected
e.g. CTK 13.2 plus CTK 13.0 is rejected

In other words, v1 is not currently trying to encode a more permissive CTK-to-CTK compatibility matrix.

It is simply requiring:

exact CTK major.minor alignment

This will probably need to be releaxed/refined.

6. Driver libraries are treated specially

Driver libraries such as cuda, nvml, and nvcudla are treated as compatibility-neutral in this model:

they do not need CTK metadata
they do not anchor the process-wide CTK line

That is intentional, because they come from the installed display driver rather than from a CUDA Toolkit line.

7. The current driver guard rail is intentionally coarse

At the moment, the driver check is:

driver_major >= ctk_major

So if the driver reports a CUDA 13.x line, then CTK 13.x is allowed, but CTK 14.x is rejected.

This is intentionally conservative and simpler than trying to encode more detailed minor-version driver/toolkit compatibility rules at this stage.

8. Optional constraints can also be imposed

CompatibilityGuardRails also supports optional explicit constraints such as:

ctk_major
ctk_minor

So callers can say, in effect:

only accept results from CTK 13
or only accept results from CTK 13.2

These explicit constraints are checked in addition to the process-wide compatibility checks.

9. Metadata insufficiency is handled separately from true incompatibility

This is an important rollout detail.

The three environment-variable modes are:

_COMPATIBILITY_GUARD_RAILS_ENV_VAR = "CUDA_PATHFINDER_COMPATIBILITY_GUARD_RAILS"
_COMPATIBILITY_GUARD_RAILS_MODES = ("off", "best_effort", "strict")

And the behaviors are:

off:
- bypass the guard rails entirely
best_effort:
- run the checks
- if the answer is insufficient metadata, fall back to the raw pathfinder behavior
- if the answer is actual incompatibility, still raise
strict:
- raise both on incompatibility and on insufficient metadata

So best_effort is not "ignore all problems". It only softens the handling of metadata gaps.

10. Noteworthy limitation: guard rails only see what flows through pathfinder

This is a very important boundary.

The current guard rails can only check items that are actually resolved through these public pathfinder APIs.

They cannot catch a mismatch if downstream code does something like:

compute include directories independently
inject its own -I/usr/local/cuda/include
use its own CUDA_PATH / LD_LIBRARY_PATH / default compiler search rules
otherwise bypass pathfinder for part of the resolution process

So I would phrase the current scope this way:

PR 1977 can catch inconsistencies among artifacts resolved through pathfinder
it does not guarantee correctness for artifacts that downstream code resolves independently

What PR 1977 does not do

I think it is important to say this explicitly.

PR 1977 does not implement the full original scoped-search idea from issue 1038.

In particular, it does not yet enforce:

"if the pivot library came from site-packages, later searches must also come from site-packages"
"if the pivot library came from Conda, later searches must also come from Conda"
"if the pivot library came from /usr/local/cuda-13.2, later searches must stay under that same root"

Instead, PR 1977 says:

resolve items using the existing search logic
then check whether the results are compatible with the already-established process state

That is a meaningful and useful subset of issue 1038, but it is not the whole original design.

Why I still think this is the right first step

I think this split is justified for two reasons.

1. It already catches real, confusing failures

The guard rails already catch real-world mixed-environment situations that would otherwise fail deep in the stack with confusing runtime errors.

That is the practical value proposition of PR 1977.

2. It gives us a simpler architecture to iterate on

PR 1977 establishes the overall CompatibilityGuardRails architecture:

process-wide state
item normalization
metadata derivation
incremental checking as new items are resolved

I expect the exact nuances of the compatibility checks will need a few rounds of refinement based on user feedback.

That seems much more manageable than trying to simultaneously solve:

full search scoping
full origin consistency
and evolving compatibility policy

all in one step.

Current real-life testing beyond mocks

Beyond extensive mock-based testing to lock in behaviors, the current real-life testing includes:

CI coverage across cuda-bindings and cuda-core
CI coverage that exercises both "local CTK" and "wheel" situations
additional manual testing across:
- Conda Linux
- Conda Windows
- CTK 12.9.1
- CTK 13.2.1

See:

manual Conda Linux/Windows testing notes

Rollout strategy

At the moment we have:

def _compatibility_guard_rails_mode() -> str:
    value = os.environ.get(_COMPATIBILITY_GUARD_RAILS_ENV_VAR)
    if not value:
        return "strict"
    if value in _COMPATIBILITY_GUARD_RAILS_MODES:
        return value

So current CI behavior is effectively:

unset / empty => strict

That is useful for testing, but I do not think that necessarily implies the same default should be used immediately in a release.

My current rollout thought is:

first release the functionality with default mode off
gather experience in opt-in mode
then switch the default to best_effort
after more experience, consider making strict the default

That would let us land the architecture and diagnostics first, without immediately changing default behavior for all users.

Bottom line

My current position is:

Issue 1038 originally covered both scoped search and compatibility checking
PR 1977 intentionally tackles the compatibility-checking half first
this includes a process-wide singleton model because dynamic library loading is process-global and irreversible
the current v1 semantics are intentionally conservative:
- exact CTK major.minor match for CTK-backed items
- coarse driver-major guard rail
- explicit handling of insufficient metadata
this already catches real-world mixed-environment failures with much better diagnostics
it does not yet implement full same-origin / same-package-system scoped search

So my current framing would be:

PR 1977 is a substantial, useful subset of issue 1038
it establishes the right architecture for compatibility guard rails
and it leaves open the possibility of revisiting fuller scoped-search behavior later, if we still think that is worth the added search complexity

Issue 1038 / PR 1977 Minor Version Compatibility Research

This note collects the CUDA minor-version-compatibility research that will inform the next guard-rails data model and decision-matrix design. It focuses on the official NVIDIA documentation, then synthesizes what those docs imply for cuda.pathfinder compatibility guard rails, especially around nvrtc, nvJitLink, nvvm, and CUDA Toolkit (CTK) headers.

Docs

CUDA Compatibility
- Top-level compatibility guide covering backward, minor-version, and forward compatibility.
Why CUDA Compatibility
- High-level framing of the three compatibility modes and their intended use.
Minor Version Compatibility
- The main official statement of CUDA minor-version compatibility and its caveats.
Forward Compatibility
- Describes the separate forward-compatibility mechanism and the cuda-compat-* packages.
Frequently Asked Questions
- Clarifies how backward compatibility, new APIs, PTX, and containers fit into the picture.
CUDA Toolkit Release Notes
- Provides the driver-range table for minor-version compatibility and the per-CTK packaged driver versions.
CUDA C++ Programming Guide: Binary Compatibility
- Background material for PTX and binary-compatibility caveats.
libNVVM API
- Documents nvvmVersion(), nvvmIRVersion(), nvvmLLVMVersion(), and NVVM_ERROR_IR_VERSION_MISMATCH.
NVVM IR Specification (current)
- Documents the current NVVM IR versions and dialects.
NVVM IR Specification (CUDA 11.8)
- Useful historical reference point showing NVVM IR 1.11.
NVVM IR Specification (CUDA 11.0)
- Useful historical reference point showing NVVM IR 1.5.
nvJitLink
- Documents JIT-link compatibility, including specific rules for LTO inputs.
PTX Compiler API
- Documents CTK-side PTX compilation, including why it can help when the installed driver's PTX JIT is older.

What NVIDIA says

NVIDIA's compatibility model distinguishes three different cases, using "backward" / "forward" terminology for the Driver-vs-CTK compatibility question:

Backward compatibility: newer drivers can run applications built against older CTKs.
Minor-version compatibility: starting with CUDA 11, an older driver from the same major family can run applications built against a newer CTK from that same major family, subject to caveats (see below).
Forward compatibility: a separate mechanism, using cuda-compat-*, for some older-driver / newer-CTK combinations across major families on supported Linux systems and hardware.

For CUDA 11 and later, NVIDIA's published minor-version-compatibility driver ranges are:

CUDA Toolkit family	Minor-version-compatibility driver range
13.x	`>= 580`
12.x	`>= 525` and `< 580`
11.x	`>= 450` and `< 525`

The same docs also say that newer drivers remain supported through ordinary backward compatibility, so the upper bound is specifically the range in which "older driver, newer CTK" minor-version compatibility is the relevant mechanism.

NVIDIA gives three explicit caveats for minor-version compatibility:

Limited feature set: some APIs or features introduced in a newer CTK require a newer driver and can fail with cudaErrorCallRequiresNewerDriver.
PTX caveat: applications using PTX will see runtime issues on older drivers.
Architecture-targeting caveat: minor-version compatibility requires an explicit architecture target such as nvcc -arch=sm_xx.

NVIDIA also says that libraries which support minor-version compatibility can be deployed across different library versions within the same major family, but library interdependencies still matter. The docs specifically call out cuDNN and cuBLAS as an example of versioned library interdependencies that should not be ignored.

The compatibility docs contain one explicit library-family statement that is especially relevant here: a figure caption says NVRTC supports minor-version compatibility from CUDA 11.3 onwards. That is much more specific than the general CUDA-runtime statement and suggests that not every compiler-adjacent component should be assumed to follow the same rule in the same way.

The forward-compatibility docs make clear that forward compatibility is a different mechanism, with narrower applicability:

it is mainly for Linux systems with supported datacenter GPUs, selected NGC Server Ready RTX systems, and Jetson boards;
it requires explicit cuda-compat-* libraries rather than happening automatically; and
those packages include PTX/LTO-relevant user-mode driver components such as:
- libnvidia-ptxjitcompiler.so.* for PTX JIT, and
- libnvidia-nvvm.so.* for JIT LTO (CUDA 11.5 and later).

On the nvvm side, NVIDIA's docs say:

libNVVM compiles NVVM IR to PTX.
libNVVM exposes version-query APIs:
- nvvmVersion()
- nvvmIRVersion()
- nvvmLLVMVersion()
libNVVM can fail with NVVM_ERROR_IR_VERSION_MISMATCH.

The NVVM IR specs show meaningful version changes across CTK eras:

CUDA 11.0 documents NVVM IR 1.5.
CUDA 11.8 documents NVVM IR 1.11.
current docs describe NVVM IR 2.0 and explicitly say that NVVM IR 2.0 is incompatible with 1.11.

Current NVVM IR docs also add a modern dialect based on LLVM 21.1.0, only for Blackwell-and-later architectures (compute_100 and newer). So nvvm compatibility is not just "same CTK major/minor or not"; IR version and dialect matter too.

Unlike the NVRTC figure, I did not find a CUDA-compatibility-page statement that says "libNVVM supports minor-version compatibility from X onwards." NVIDIA's published compatibility material appears to treat nvvm more indirectly, via PTX/LTO and forward-compatibility components, than it treats NVRTC.

On the nvJitLink side, NVIDIA is much more explicit. The nvJitLink docs say:

nvJitLink is compatible across minor versions within a release family.
the nvJitLink library version must be >= the maximum version of the inputs.
compatibility is not generally guaranteed across majors for LTO inputs.
linking across major versions works for ELF and PTX inputs, but does not work with LTOIR inputs.
if using LTO, compatibility is only guaranteed within a major release.

The PTX Compiler API docs reinforce the PTX caveat from another angle: CTK-side PTX compilation can support newer PTX versions even when the installed driver's PTX JIT support is older. That is a strong signal that PTX-sensitive flows deserve their own compatibility treatment rather than being lumped in with ordinary runtime-library loading.

Key implications

The single most important takeaway is that NVIDIA's official minor-version-compatibility rule is fundamentally a display-driver-release rule, not a "cuDriverGetVersion() must be at least the CTK version" rule.

NVIDIA's own example uses a 450.80.02 driver that reports CUDA 11.0 while successfully running a CUDA 11.1 application. That means a check of the form:

driver CUDA version must be >= CTK version

is stricter than NVIDIA's documented minor-version-compatibility contract.

That does not make the stricter rule useless. It just means it is a different policy:

it may be a reasonable conservative guard rail in some contexts,
but it is not the same thing as official NVIDIA minor-version compatibility.

This leads to a second key takeaway: driver/CTK compatibility and cross-artifact CTK coherence are different questions.

The official docs are primarily about whether an application built against a newer CTK can run on a system with an older driver. They are not a blanket endorsement of mixing arbitrary CTK-minor components at build or JIT time. In particular, they do not justify treating mismatched headers, nvcc, nvrtc, nvvm, or nvJitLink as interchangeable just because the driver happens to be minor-version-compatible.

That distinction matters a lot for guard rails:

cuda.h or headers more generally are build-time / compile-time artifacts.
nvcc, nvrtc, nvvm, and nvJitLink sit on compiler or JIT paths.
many other CTK shared libraries are more like runtime libraries.

Those classes should not automatically share the same compatibility rules.

The third key takeaway is that PTX-sensitive and LTO-sensitive flows are special.

NVIDIA says this directly for PTX in the minor-version-compatibility guide. The nvJitLink and PTX Compiler API docs say it again from a tooling angle: newer CTK-side JIT/link/compile components can offer capabilities that older driver-side components do not.

That means:

nvrtc is not just a library load; it is often a PTX-producing runtime compiler.
nvvm is not just a library load; it is an IR-to-PTX compiler with its own IR-version boundaries.
nvJitLink is not just a library load; it has input-version and LTO-specific compatibility rules of its own.

nvvm in particular deserves special consideration. The docs show two separate reasons:

IR compatibility: NVIDIA explicitly documents NVVM IR version changes and incompatibilities.
PTX/LTO downstream sensitivity: libNVVM produces PTX, and JIT LTO is important enough that forward-compatibility packages ship libnvidia-nvvm.so.* specifically for that path.

So while nvvm clearly belongs in the CTK ecosystem, it should not be modeled as a generic "same-major library that minor-version compatibility probably covers."

Finally, forward compatibility should be kept conceptually separate. It is tempting to say "older driver plus newer CTK may still work somehow," but NVIDIA's actual forward-compatibility mechanism is narrower, more explicit, and more environment-dependent than ordinary minor-version compatibility. Guard rails should not silently assume it.

What this means for guard rails

The current research points toward a guard-rails design with separate policy dimensions, rather than a single notion of "compatible" or "incompatible."

At minimum, the design should distinguish:

Scoped CTK coherence
- Are the resolved CTK artifacts mutually coherent with each other?
- Example: headers, nvcc, nvrtc, nvvm, nvJitLink, and CTK shared libraries resolving to the same CTK minor is a stronger and more local property than driver compatibility.
Driver/CTK compatibility
- Is the driver compatible with the resolved CTK by:
  - backward compatibility,
  - minor-version compatibility,
  - forward compatibility,
  - or not at all?
- This requires more than DriverCudaVersion; it likely needs the actual display-driver version or branch.
Compiler / JIT sensitivity
- Does the flow involve:
  - ordinary runtime library loading,
  - PTX,
  - NVVM IR,
  - or LTO IR?
- These categories have different compatibility constraints in NVIDIA's docs.

From that perspective, the conservative near-term guard-rails recommendations are:

Keep exact CTK major.minor coherence checks for same-scoped CTK artifacts by default.
- This is especially important for headers (cuda.h and friends), nvcc, nvrtc, nvvm, and nvJitLink.
- The minor-version-compatibility docs do not provide a strong basis for relaxing these checks.
Add display-driver-version data alongside DriverCudaVersion.
- Official NVIDIA minor-version compatibility is expressed in driver-release terms.
- DriverCudaVersion is still useful, but on its own it is not enough to model NVIDIA's published policy faithfully.
Treat nvrtc, nvJitLink, and nvvm as compiler/JIT-sensitive artifacts, not as ordinary runtime libraries.
- nvrtc: PTX-producing, with an explicit NVIDIA note that minor-version compatibility starts from CUDA 11.3 onward.
- nvJitLink: has explicit same-major minor compatibility rules, plus stricter LTO constraints.
- nvvm: has explicit IR-version and dialect constraints, and feeds PTX/LTO-sensitive paths.
Model nvvm conservatively unless and until the data model can represent IR version/dialect.
- The published docs support saying that nvvm needs special handling.
- They do not support saying that libNVVM generally enjoys the same simple minor-version-compatibility story as ordinary runtime components.
Do not infer forward compatibility silently.
- If forward compatibility is ever considered, it should depend on explicit evidence such as detected cuda-compat-* components and a supported environment.
Make the decision matrix answer separate questions, rather than flattening everything into one result:
- Are the resolved CTK artifacts mutually coherent?
- Is the driver compatible with the CTK by backward/minor/forward rules?
- Does this flow involve PTX, NVVM IR, or LTO, making the compatibility story stricter?

In short: the research supports a data model where cuda.h/headers, nvrtc, nvJitLink, and especially nvvm are not all treated as generic CTK artifacts. Driver minor-version compatibility is real and important, but it should be modeled separately from exact CTK coherence and separately again from PTX/NVVM/LTO-sensitive flows.

I used Claude Opus 4.7 Adaptive to fact-check the What NVIDIA says section in the previous comment. It only found one minor error that's already corrected (one version number was wrong). I also requested a rewrite to improve readability. I'm pasting the result here. Please pick and choose what seems more readable to you. The information should be exactly identical.

What NVIDIA says

This rewrite groups the relevant NVIDIA documentation by topic so each statement
sits next to the doc it came from. Items marked (verbatim) are direct
quotes from NVIDIA's pages; everything else is a paraphrase. Sources are
listed at the end of each subsection so any teammate can re-check.

1. The compatibility model: three distinct mechanisms

NVIDIA's compatibility model has three named mechanisms, and they answer
different questions:

Mechanism	Question it answers
Backward compatibility	Can a newer driver still run an application built against an older CTK?
Minor-version compatibility	Can an older driver from the same major family run an application built against a newer CTK from that same major?
Forward compatibility	Can a `cuda-compat-*` package let an old-major-family driver run a newer-major-family CTK on certain Linux systems?

Backward compatibility is implicit and "always on" for new drivers running old
CTK applications. Minor-version compatibility was introduced in CUDA 11 and is
opt-in subject to caveats (next subsection). Forward compatibility is a
separately packaged mechanism with much narrower applicability.

Sources:

Why CUDA Compatibility: https://docs.nvidia.com/deploy/cuda-compatibility/why-cuda-compatibility.html
Minor Version Compatibility: https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html
Forward Compatibility: https://docs.nvidia.com/deploy/cuda-compatibility/forward-compatibility.html

2. Minor-version compatibility: driver ranges and caveats

NVIDIA's published driver ranges for minor-version compatibility are:

CTK family	Minimum driver	Upper bound for minor-version compatibility
13.x	`>= 580`	none — newer drivers covered by backward compatibility
12.x	`>= 525`	`< 580` — newer drivers covered by backward compatibility
11.x	`>= 450`	`< 525` — newer drivers covered by backward compatibility

The upper bound is the range in which the minor-version-compatibility
mechanism is the relevant one. Past it, the same combination still works,
but via plain backward compatibility instead.

NVIDIA gives three explicit caveats. If any of them apply, minor-version
compatibility is not enough and the driver must be upgraded:

Limited feature set. Some features span the toolkit and the driver
together, and an application that uses them on an older driver may return
cudaErrorCallRequiresNewerDriver.
PTX caveat. (verbatim) "Applications that compile device code to
PTX will not work on older drivers."
Architecture-target caveat. Minor-version compatibility requires an
explicit architecture target, e.g. nvcc -arch=sm_xx.

NVIDIA also says minor-version-compatible libraries can be deployed against
different library minor versions in the same major family without
recompiling the application — but adds that libraries have interdependencies
that have to be respected. The cited example is that each cuDNN version
requires a specific cuBLAS version.

The compatibility page contains exactly one library-family figure caption
that is directly relevant to compiler-adjacent components:

(verbatim) NVRTC supports minor version compatibility from CUDA 11.3 onwards.

This is much more specific than the general CUDA-runtime statement, and it
suggests that the same simple rule should not automatically be assumed for
every other compiler-adjacent component. There is no analogous published
"libNVVM supports minor-version compatibility from X onwards" statement on
the same page.

Sources:

Minor Version Compatibility: https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html
(Table 1, the three caveats, the deployment paragraph, and Figure 3)
CUDA Toolkit Release Notes: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html

3. Forward compatibility: a separate, narrower mechanism

Forward compatibility is not the same thing as minor-version compatibility,
and the docs are explicit about the difference.

Where it applies. Forward compatibility is documented as applicable to
Linux systems with one of:

NVIDIA Data Center GPUs,
selected NGC Server Ready SKUs of RTX cards,
Jetson boards.

It is intended to let applications built against a newer CTK run on a system
whose installed Linux driver belongs to an older major release family.

How it is delivered. It does not happen automatically. It requires a
cuda-compat-<cuda_major_version>-<cuda_minor_version> package, installed
into a versioned toolkit directory and made visible through the dynamic
loader (typically via LD_LIBRARY_PATH or equivalent).

What is in the package. The cuda-compat-* package consists of:

libcuda.so.* (the CUDA driver),
libcudadebugger.so.* (CUDA 11.8 and later only),
libnvidia-nvvm.so.* (JIT LTO; CUDA 11.5 and later only),
libnvidia-ptxjitcompiler.so.* (PTX JIT compiler),
libnvidia-pkcs11*.so.* (OpenSSL support).

The two especially relevant entries for our compatibility-guard-rails work
are libnvidia-ptxjitcompiler.so.* (PTX JIT) and libnvidia-nvvm.so.*
(JIT LTO, CUDA 11.5+) — i.e. the user-mode driver components that PTX- and
LTO-sensitive flows ultimately depend on.

Sources:

Forward Compatibility: https://docs.nvidia.com/deploy/cuda-compatibility/forward-compatibility.html

4. NVRTC

NVRTC is a runtime compiler for CUDA C++; its primary output is PTX, which is
then loaded via cuModuleLoadData[Ex] or linked via nvJitLink / cuLinkAddData.
That is, NVRTC is not just a runtime library load — it is a PTX producer,
which is exactly what the PTX caveat in §2 is talking about.

The single most specific minor-version-compatibility statement NVIDIA makes
for a compiler-adjacent component is the figure caption already quoted in §2:
(verbatim) "NVRTC supports minor version compatibility from CUDA 11.3
onwards."

Sources:

NVRTC: https://docs.nvidia.com/cuda/nvrtc/index.html
Minor Version Compatibility (Figure 3): https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html

5. libNVVM and NVVM IR

libNVVM compiles NVVM IR to PTX, and exposes three version-query APIs:

nvvmVersion(major, minor) — the libNVVM library version
nvvmIRVersion(majorIR, minorIR, majorDbg, minorDbg) — the NVVM IR version (and debug-metadata version)
nvvmLLVMVersion(arch, major) — the LLVM IR version guaranteed to be supported for a given arch

It can fail with NVVM_ERROR_IR_VERSION_MISMATCH when given IR from an
incompatible producer.

NVVM IR has had non-trivial version changes across CTK eras:

CTK	NVVM IR version documented
11.0	1.5
11.8	1.11
current (CTK 13.2)	2.0

The current NVVM IR Specification is explicit about non-compatibility:
(verbatim) "The 2.0 version of NVVM IR is incompatible with the
previous version 1.11. Linking of NVVM IR Version 1.11 with 2.0 will result
in compiler error."

In addition, NVVM IR currently has two dialects:

The LLVM 7 dialect (LLVM 7.0.1) — used for pre-Blackwell targets.
The modern dialect — currently based on LLVM 21.1.0 (was 20.1.0 in
the CTK 13.0 archive of this doc, and the LLVM version stepped through
18.1.8 / 19.1.0 across earlier CTK 12.8 / 12.9 archives, so this number
is a moving target). The modern dialect only supports compute capability
compute_100 and later (Blackwell and later).

So nvvm compatibility is not just "same CTK major/minor or not"; the IR
version and the dialect both matter.

Notably, the Minor Version Compatibility page does not contain a
"libNVVM supports minor-version compatibility from X onwards" figure
analogous to the NVRTC one. NVIDIA's published material treats nvvm
more indirectly — through PTX/LTO and through the forward-compatibility
package contents in §3 — than it treats NVRTC.

Sources:

libNVVM API: https://docs.nvidia.com/cuda/libnvvm-api/
NVVM IR Specification (current, release 13.2): https://docs.nvidia.com/cuda/nvvm-ir-spec/index.html
NVVM IR Specification (CUDA 11.0 archive): https://docs.nvidia.com/cuda/archive/11.0_GA/nvvm-ir-spec/index.html
NVVM IR Specification (CUDA 11.8 archive): https://docs.nvidia.com/cuda/archive/11.8.0/nvvm-ir-spec/index.html

6. nvJitLink

The nvJitLink docs are unusually explicit about compatibility, and almost
everything they say survives verbatim:

(verbatim) "The nvJitLink library is compatible across minor versions
in a release, but may not be compatible across major versions."
(verbatim) "The library version itself must be >= the maximum
version of the inputs."
(verbatim) "Linking across major versions (like 11.x with 12.x) works
for ELF and PTX inputs, but does not work with LTOIR inputs."
(verbatim) "If using LTO, then compatibility is only guaranteed
within a major release."

So nvJitLink has same-major minor-version compatibility plus an additional
LTO-input restriction that tightens the rule.

Sources:

nvJitLink: https://docs.nvidia.com/cuda/nvjitlink/index.html#introduction

7. PTX Compiler API

The PTX Compiler API docs reinforce the PTX caveat from a tooling angle.
They say (paraphrased): the PTX Compiler APIs allow runtime compilation for
the latest PTX version supported by the CUDA Toolkit release; this support
may not be available in the PTX JIT compiler in the installed CUDA driver
if that driver is older.

That is a strong, NVIDIA-stated signal that PTX-sensitive flows deserve their
own compatibility treatment rather than being lumped in with ordinary
runtime-library loading.

Sources:

PTX Compiler API: https://docs.nvidia.com/cuda/ptx-compiler-api/

8. Background reference

The CUDA C++ Programming Guide section on binary compatibility is the
underlying reference for the PTX and binary-compatibility caveats above.
It does not introduce additional rules on top of what is already stated
in the compatibility, NVRTC, NVVM, nvJitLink, and PTX-Compiler-API docs,
but it is the canonical place those caveats are eventually cross-referenced
to.

Sources:

CUDA C++ Programming Guide — Binary Compatibility:
https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#binary-compatibility
CUDA Compatibility — FAQ:
https://docs.nvidia.com/deploy/cuda-compatibility/frequently-asked-questions.html

Granular CTK and Pipeline Compatibility Checks

This note proposes the next shape of the cuda.pathfinder compatibility guard rails.

The goal is to move beyond one global rule that all resolved CUDA Toolkit (CTK) artifacts must have the same major/minor version. That rule is useful as an early guard rail, but it is stricter than what we ultimately want. The better model is to enforce exact CTK coherence where artifacts are tightly coupled, while allowing documented and plausible compatibility across independent artifacts and explicit compiler/JIT pipelines.

The guiding principle is:

Allow as much as NVIDIA's compatibility model supports, while reliably rejecting combinations we know cannot work.

Separate Compatibility Questions

The guard rails should answer several separate questions instead of flattening everything into one "compatible or incompatible" decision.

Are directly coupled CTK artifacts mutually coherent?
Is each resolved CTK artifact compatible with the installed driver?
Is a compiler or JIT pipeline using artifacts in a documented compatible order?
Is there enough metadata to answer the relevant question confidently?

These questions need different rules.

For example, two libraries that are linked together in the same ELF dependency component are not the same kind of relationship as libnvrtc.so producing PTX or LTOIR that is later consumed by libnvJitLink.so. The first is a dynamic-link dependency relation; the second is a semantic data-flow relation. Both matter, but they should not be modeled as the same thing.

Dynamic-Link Components

The generated dependency graph in Pathfinder13Venv_graphviz.svg below gives useful evidence about dynamic-link coupling between CUDA libraries.

Within a connected dynamic-link component, the conservative and readable rule should be:

All CTK-packaged libraries in the same dynamic-link component must resolve to the same CTK major/minor version.

This keeps exact CTK coherence where the libraries are loaded into the same ABI island and directly depend on one another.

Examples of components visible in the current graph include:

cusolver, cusparse, cublas, cublasLt, cutensor, nccl, cudart, and nvJitLink
the npp* libraries
cufft, cufftMp, and nvshmem_host
mathdx and nvrtc
cufile and cufile_rdma

This rule is deliberately local. It should not imply that every CTK library loaded in the process must have the same minor version.

Independent CTK Artifacts

For CTK artifacts that are not in the same dynamic-link component and do not have another known tight relationship, exact minor-version equality should not be required by default.

Instead, each artifact should be checked against the driver compatibility policy, plus any role-specific constraints.

This matters for isolated libraries in the graph such as libnvvm, libnvfatbin, libcupti, libnvjpeg, and libcurand. Loading one of these from CTK 13.2 should not automatically make an otherwise independent CTK 13.1 artifact invalid solely because their minor versions differ.

The guard rail should still require enough metadata to know what CTK version each artifact belongs to. The relaxation is about global minor equality, not about accepting unknown provenance.

Driver Compatibility

Driver compatibility should be checked separately from CTK artifact coherence.

The current strict rule, roughly "driver CUDA version must be at least the CTK version," is conservative but does not fully represent NVIDIA's documented minor-version compatibility model. NVIDIA's minor-version compatibility policy is expressed in terms of display driver release ranges, not only cuDriverGetVersion().

The next model should therefore treat driver compatibility as its own decision:

Newer drivers should accept older CTK artifacts through ordinary backward compatibility.
Older drivers in the documented same-major range may accept newer CTK artifacts through minor-version compatibility, subject to NVIDIA's caveats.
Forward compatibility should remain explicit and opt-in, not silently inferred.

This check should apply per relevant CTK artifact or per selected CTK version, but it should not force unrelated CTK artifacts to have the same minor version.

Compiler and JIT Pipelines

Compiler and JIT components need additional rules because compatibility depends on the artifacts being produced and consumed.

The important example for the next guard-rails iteration is:

A newer libnvJitLink.so may process artifacts produced by an older libnvrtc.so when NVIDIA documents that pipeline as supported.

This should not be rejected merely because nvrtc and nvJitLink have different CTK minor versions.

The key distinction is that nvrtc and nvJitLink are not necessarily dynamically linked to each other. Their compatibility relation often comes from user data flow:

nvrtc produces PTX, CUBIN, or LTOIR.
Some later step passes that artifact to nvJitLink.
The compatibility question depends on the artifact kind and versions.

For LTOIR inputs, NVIDIA documents a stricter relation:

nvJitLink must be at least as new as the producer of the LTOIR.
LTO compatibility is within a major release family.
Cross-major LTOIR compatibility should not be assumed.

For PTX and ELF inputs, the compatibility story is broader, but still tied to NVIDIA's documented caveats around PTX, driver JIT support, architecture targeting, and CTK-side tooling.

Proposed Decision Tree

At a high level, registering a newly resolved artifact should look like this:

def check_new_item(state, item):
    require_supported_packaging(item)
    require_ctk_metadata_if_ctk_packaged(item)
    enforce_user_ctk_version_constraint(item)
    check_driver_compatibility(item)

    for prior in state.resolved_items:
        relation = classify_relation(prior, item)

        if relation == "same_dynamic_link_component":
            require_exact_ctk_major_minor(prior, item)

        elif relation == "known_companion_artifacts":
            require_exact_ctk_major_minor(prior, item)

        elif relation == "declared_pipeline":
            check_pipeline_compatibility(prior, item)

        elif relation == "independent_ctk_artifacts":
            pass

    state.remember(item)

The relation classifier is the key new abstraction. It should classify pairs of artifacts by the strongest known relationship between them:

Same dynamic-link component
Known companion artifacts
Explicit compiler/JIT pipeline
Independent CTK artifacts
Unknown relationship

Unknown relationships should be handled conservatively at first. In strict mode, unknown-but-relevant metadata may remain an error; in warning or best-effort modes, it can be surfaced or bypassed according to the existing mode semantics.

Pipeline Decision Tree

Pipeline checks should be explicit and narrow. They should only apply when pathfinder knows enough about the producer, consumer, and artifact kind.

def check_pipeline_compatibility(producer, consumer, artifact_kind):
    if producer.name == "nvrtc" and consumer.name == "nvJitLink":
        if artifact_kind == "ltoir":
            require_same_ctk_major(producer, consumer)
            require_version_at_least(consumer, producer)
            return

        if artifact_kind in {"ptx", "elf", "cubin"}:
            allow_if_driver_and_architecture_constraints_are_not_known_bad()
            return

        require_more_metadata_or_warn()
        return

    if producer.name == "nvvm":
        require_known_supported_nvvm_ir_relation_or_exact_ctk_match()
        return

    require_more_metadata_or_warn()

This avoids weakening general CTK coherence rules just because a particular producer/consumer pipeline may support mixed minor versions.

Practical First Implementation

A practical first implementation can be intentionally conservative:

Replace the single process-wide "all CTK artifacts must exactly match" anchor with pairwise relation checks.
Encode the known dynamic-link components from the generated graph as data.
Require exact CTK major/minor within each dynamic-link component.
Treat artifacts outside those components as independent unless a known companion or pipeline rule applies.
Keep exact CTK matching for headers, nvcc, and compile-scope companion artifacts.
Add an explicit nvrtc to nvJitLink pipeline rule that allows newer nvJitLink to consume older nvrtc output when the artifact kind is supported.
Keep driver compatibility as a separate check.

This gives a better day-one balance:

It still catches concrete CTK-mixing problems inside tightly coupled library groups.
It stops rejecting independent CTK artifacts solely because their minor versions differ.
It makes room for documented JIT compatibility, especially nvrtc to nvJitLink.
It keeps forward compatibility and unknown metadata explicit rather than silently assuming success.

Summary

The next guard-rails model should be relation-based rather than globally anchored.

Exact CTK major/minor coherence remains the right default for directly coupled artifacts: dynamic-link components, headers with compile tools, and known companion libraries. Independent CTK artifacts should be checked against the driver and their own role-specific constraints, but should not force global minor-version equality. JIT and compiler pipelines should be modeled explicitly, with rules based on producer, consumer, artifact kind, and documented NVIDIA compatibility guarantees.

That structure lets cuda.pathfinder be less restrictive where NVIDIA supports compatibility, while still rejecting combinations that are tightly coupled or known not to work.

Pathfinder13Venv CUDA library dependency graph

The SVG file was generated with this pair of scripts:

Command used:

generate_ctk_ldd_graphs.sh /wrk/forked/cuda-python/Pathfinder13Venv/lib/python3.12/site-packages Pathfinder13Venv

The venv was build with:

cd cuda_pathfinder/
pip install -v -e . --group test-cu13

$ /wrk/forked/cuda-python/Pathfinder13Venv/bin/pip list
Package                  Version                Editable project location
------------------------ ---------------------- ---------------------------------------
cuda-bindings            13.2.0
cuda-pathfinder          1.5.5.dev28+g2ea4b26f7 /wrk/forked/cuda-python/cuda_pathfinder
cuda-python              13.2.0
cuda-toolkit             13.2.1
cutensor-cu13            2.6.0
iniconfig                2.3.0
networkx                 3.6.1
numpy                    2.4.4
nvidia-cublas            13.4.0.1
nvidia-cublasmp-cu13     0.8.1.2360
nvidia-cuda-cccl         13.2.75
nvidia-cuda-crt          13.2.78
nvidia-cuda-cupti        13.2.75
nvidia-cuda-nvcc         13.2.78
nvidia-cuda-nvrtc        13.2.78
nvidia-cuda-profiler-api 13.2.75
nvidia-cuda-runtime      13.2.75
nvidia-cudss-cu13        0.7.1.6
nvidia-cufft             12.2.0.46
nvidia-cufftmp-cu13      12.1.3.2
nvidia-cufile            1.17.1.22
nvidia-curand            10.4.2.55
nvidia-cusolver          12.2.0.1
nvidia-cusolvermp-cu13   0.8.0.3126
nvidia-cusparse          12.7.10.1
nvidia-cusparselt-cu13   0.9.1
nvidia-cutlass           4.2.0.0
nvidia-libmathdx-cu13    0.3.2.6
nvidia-nccl-cu13         2.30.4
nvidia-npp               13.1.0.48
nvidia-nvfatbin          13.2.78
nvidia-nvjitlink         13.2.78
nvidia-nvjpeg            13.1.0.48
nvidia-nvshmem-cu13      3.6.5
nvidia-nvvm              13.2.78
packaging                26.2
pip                      26.1
pluggy                   1.6.0
pydot                    4.0.1
Pygments                 2.20.0
pyparsing                3.3.2
pytest                   9.0.3
pytest-mock              3.15.1
pytest-randomly          4.1.0
pytest-repeat            0.9.4
scipy                    1.17.1
six                      1.17.0
treelib                  1.8.0

`CUDA_PATHFINDER_DRIVER_COMPATIBILITY` shape

This note captures a proposed shape for a forward-compatibility escape hatch.

Goal

Keep the default guard rails strict and helpful, while avoiding the "pathfinder is in my way" experience for the rare Linux setups that intentionally rely on NVIDIA forward compatibility.

The escape hatch should be:

narrow,
explicit,
Linux-only,
and advertised in error messages when it is actually relevant.

Recommended shape

Keep _COMPATIBILITY_GUARD_RAILS_MODES unchanged:

off
best_effort
strict

Add a separate, orthogonal env var:

CUDA_PATHFINDER_DRIVER_COMPATIBILITY=default|assume_forward_compatibility

Preferred interpretation:

default
- No special assumption.
- Driver-vs-CTK checks behave normally.
assume_forward_compatibility
- On supported Linux systems, suppress the specific driver-vs-CTK incompatibility errors that arise from not modeling NVIDIA forward compatibility.

Why this should not be another mode

off, best_effort, and strict describe enforcement / fallback behavior.

assume_forward_compatibility would describe a semantic assumption about one specific compatibility question.

Those are different axes.

If assume_forward_compatibility were added to _COMPATIBILITY_GUARD_RAILS_MODES, people would immediately want combinations like:

strict + assume_forward_compatibility
best_effort + assume_forward_compatibility

That is a strong sign that it should be modeled as a separate knob.

Keeping the axes separate also makes the behavior easier to explain:

mode answers: how aggressively should pathfinder enforce the guard rails?
driver-compatibility override answers: should pathfinder assume Linux forward compatibility for the driver-vs-CTK relation?

Scope of the override

The override should affect only driver-vs-CTK compatibility checks.

It should not relax:

exact CTK coherence checks,
header/library mismatch checks,
cuda.h vs runtime/compiler/JIT mismatches,
nvrtc / nvvm / nvJitLink scoped-consistency checks,
PTX/NVVM/LTO-sensitive coherence checks.

In other words, the override means:

Do not block me on the driver-vs-CTK relation, because this Linux system is intentionally relying on NVIDIA forward compatibility.

It does not mean:

Ignore general CTK inconsistency problems.

Linux-only behavior

This should be Linux-only from day one.

Recommended behavior:

On Linux:
- default and assume_forward_compatibility are accepted.
On non-Linux:
- setting CUDA_PATHFINDER_DRIVER_COMPATIBILITY=assume_forward_compatibility should raise a clear error rather than silently doing nothing.

That keeps the behavior honest and prevents cargo-cult configuration on unsupported platforms.

Error-message behavior

Error conditions that can be resolved by the env var should systematically advertise it.

Recommended rule:

When pathfinder is about to fail solely because of a driver-vs-CTK incompatibility,
and the failure would be bypassed by CUDA_PATHFINDER_DRIVER_COMPATIBILITY=assume_forward_compatibility,
append a standard hint.

Suggested wording:

On supported Linux systems that intentionally rely on NVIDIA forward compatibility (cuda-compat-*), set CUDA_PATHFINDER_DRIVER_COMPATIBILITY=assume_forward_compatibility to bypass this driver-vs-CTK check. This does not relax CTK-coherence checks between headers, libraries, and compiler/JIT components.

Important constraints:

The hint should come from one central helper so the wording stays consistent.
The hint should only be shown when it is genuinely actionable.
The hint should not be shown for ordinary CTK-coherence failures.

Recommendation

For an initial implementation:

Keep _COMPATIBILITY_GUARD_RAILS_MODES as off | best_effort | strict.
Add CUDA_PATHFINDER_DRIVER_COMPATIBILITY=default|assume_forward_compatibility.
Make the override Linux-only.
Apply it only to driver-vs-CTK compatibility failures.
Systematically advertise it in relevant strict-mode error messages.

This gives users an escape hatch immediately, without weakening the main guard-rails model or overcomplicating the first version.

Future extensions

This shape leaves room for future refinements without changing the conceptual model, for example:

detecting explicit cuda-compat-* evidence instead of relying only on an override,
expanding CUDA_PATHFINDER_DRIVER_COMPATIBILITY with additional values,
distinguishing between "assume forward compatibility" and "detected forward compatibility."

For now, though, the narrow Linux-only assumption override seems like the right balance.

`warning_only` shape

This note captures a proposed shape for:

CUDA_PATHFINDER_COMPATIBILITY_GUARD_RAILS=warning_only

The intent is to provide an escape hatch with visibility: pathfinder should keep working, but users should still see what guard-rails checks are being ignored.

Core idea

warning_only should behave like strict in terms of what gets checked, but like a fallback mode in terms of what happens after a failure:

run the same compatibility checks,
generate the same messages,
downgrade failures to warnings,
then fall back to raw pathfinder behavior.

This gives users a way to keep moving without losing visibility into guard-rails findings.

Why this belongs in `_COMPATIBILITY_GUARD_RAILS_MODES`

Unlike assume_forward_compatibility, warning_only changes the enforcement style, not the compatibility semantics.

That makes it a natural fit for:

off
best_effort
warning_only
strict

By contrast, assume_forward_compatibility is a semantic assumption about one specific compatibility question and should remain a separate, orthogonal knob.

Recommended mode ladder

off
- Do raw pathfinder behavior, no guard rails.
best_effort
- Enforce guard rails, but fall back on insufficient metadata.
warning_only
- Enforce guard rails, but downgrade both incompatibility and insufficient-metadata failures to warnings, then fall back.
strict
- Enforce guard rails and raise.

This produces a useful progression:

off = ignore everything
best_effort = ignore only unknowns
warning_only = ignore everything, but visibly
strict = ignore nothing

Why this seems valuable

warning_only is a strong rollout mode:

users are not blocked,
users still see exactly what pathfinder thinks is wrong,
teams can gauge real-world impact before tightening behavior,
and the mode provides a more helpful escape hatch than either silent fallback or a hard stop.

It is also a better user experience for rare-but-real cases where strict enforcement would be perceived as pathfinder getting in the way.

Recommended semantics

warning_only should:

run the same checks as strict,
preserve the same distinction between compatibility failures and insufficient metadata,
emit warnings using the same core message text that strict would raise,
then use the raw fallback result.

Important constraint:

warning_only should not change what counts as compatible or incompatible.
It should only change how failures are surfaced.

Relationship to `assume_forward_compatibility`

These two knobs are complementary, not redundant:

warning_only means:
- "Do not block me on any guard-rails failure, but show me what you are ignoring."
assume_forward_compatibility means:
- "This specific driver-vs-CTK incompatibility is expected on this Linux system; stay strict about everything else."

So:

warning_only belongs in _COMPATIBILITY_GUARD_RAILS_MODES
assume_forward_compatibility should stay separate

Warning mechanism

Recommended implementation details:

use Python warnings,
define a custom warning category such as CompatibilityGuardRailsWarning,
reuse the exact strict-mode failure text in the emitted warning,
and keep warning construction centralized so the messaging stays consistent.

This keeps warning_only and strict aligned: the only difference is whether the message is raised as an exception or emitted as a warning.

Interaction with narrower escape hatches

When the warning corresponds to a failure that has a narrower fix, the warning should still advertise that narrower fix.

For example, a driver-vs-CTK failure that could be intentionally bypassed via:

CUDA_PATHFINDER_DRIVER_COMPATIBILITY=assume_forward_compatibility

should still say so in the warning text.

That way:

warning_only provides a broad escape hatch,
but users still discover the more targeted fix when one exists.

Day-one recommendation

For an initial implementation:

Add warning_only to _COMPATIBILITY_GUARD_RAILS_MODES.
Make it run the same checks as strict.
Downgrade both incompatibility and insufficient-metadata failures to warnings.
Fall back to the raw pathfinder result after warning.
Use a custom warning category and centralized message helper.

This seems like a strong day-one addition because it improves rollout safety without weakening the conceptual guard-rails model.

Generated with Cursor GPT-5.4 Extra High Fast

Granular Guard Rails Roadmap

Current code under PR #1977 hard-codes a single global rule in cuda_pathfinder/cuda/pathfinder/_compatibility_guard_rails.py:

if item1.ctk_version != item2.ctk_version:
    ... "v1 requires an exact CTK major.minor match."

CompatibilityGuardRails._register_and_check() then compares each new CTK-backed item against a single process-wide CTK anchor. The implementation plan below replaces that with a relation-based engine.

Target Shape

flowchart TD
  resolvedItem[ResolvedItem] --> driverPolicy[DriverVsCtkPolicy]
  resolvedItem --> relationClassifier[RelationClassifier]
  relationClassifier --> sameComponent[SameDynamicLinkComponentOrCompanion]
  relationClassifier --> independent[IndependentArtifact]
  relationClassifier --> pipeline[DeclaredPipeline]
  sameComponent --> exactMatch[RequireExactCtkMajorMinor]
  independent --> noLockstep[NoGlobalMinorLockstep]
  pipeline --> pipelineRules[ApplyNvrtcNvJitLinkOrNvvmRules]

Milestones

1. Split the policy surface before changing behavior

Refactor cuda_pathfinder/cuda/pathfinder/_compatibility_guard_rails.py so CTK-to-CTK coherence, driver-vs-CTK compatibility, and pipeline checks are separate internal decisions instead of one compatibility_check() branch.
Keep current behavior initially, but replace the single-anchor mental model with a relation-classifier hook and pairwise evaluation over remembered items.
This de-risks the later behavior change and gives one place to centralize error-message text.

2. Encode authored relation metadata from the graph research

Use the Pathfinder13Venv graph outputs as design input, then codify the resulting connected components in authored Python metadata rather than reading .gv files at runtime.
Extend cuda_pathfinder/cuda/pathfinder/_dynamic_libs/descriptor_catalog.py so each dynamic library carries enough authored metadata to classify same-component relationships; the existing dependencies= edges are the best starting point.
Extend companion metadata for non-dynamic artifacts in cuda_pathfinder/cuda/pathfinder/_headers/header_descriptor_catalog.py and the binary/static-lib surfaces so headers, nvcc, static libs, and bitcode libs can participate in the same relation model.
Add catalog-level tests so the authored components stay aligned with the intended clusters.

3. Replace the global exact-match anchor with component-local coherence checks

Change cuda_pathfinder/cuda/pathfinder/_compatibility_guard_rails.py so exact CTK major.minor matching is required only for:
- items in the same dynamic-link component,
- known companion artifacts,
- compile-scope pairings such as headers plus toolchain companions.
Remove the assumption that every CTK-backed item in the process must share one CTK minor version.
This milestone should already stop rejecting independent nvrtc and nvJitLink loads merely because their CTK minors differ.

4. Add a real driver-policy layer and the new forward-compatibility knob

Introduce CUDA_PATHFINDER_DRIVER_COMPATIBILITY=default|assume_forward_compatibility in cuda_pathfinder/cuda/pathfinder/_process_wide_compatibility_guard_rails.py.
Make it Linux-only, fail explicitly on non-Linux, and apply it only to driver-vs-CTK failures.
Centralize the strict-mode hint text there so future rollout modes can reuse the same messaging.
Keep this orthogonal to CUDA_PATHFINDER_COMPATIBILITY_GUARD_RAILS; do not introduce another guard-rails mode for forward compatibility.

5. Upgrade driver metadata beyond `cuDriverGetVersion()`

Add or extend driver utilities near cuda_pathfinder/cuda/pathfinder/_utils/driver_info.py so pathfinder can reason about display-driver release ranges, not only the CUDA-facing encoded version.
Use that data to classify backward compatibility, minor-version compatibility, and explicit forward-compatibility override cases as a separate driver decision.
Keep DriverCudaVersion because it is still useful, but stop treating it as the full driver-compatibility story.
Use commit 4eade17 archived branch in the archive/2026-04-25+110553_pr1936_pathfinder_with_compatibility_checks_v0 branch as starting point for this step.

6. Add explicit pipeline-context checks for compiler/JIT flows

Introduce a minimal pipeline-context surface so pathfinder can know producer, consumer, and artifact kind when that information exists.
Then add explicit rules for nvrtc -> nvJitLink and nvvm in cuda_pathfinder/cuda/pathfinder/_compatibility_guard_rails.py:
- LTOIR: nvJitLink same major and >= producer version.
- PTX/ELF/CUBIN: broader allow rules, gated by known NVIDIA caveats.
- nvvm: remain conservative until the model can represent NVVM IR version/dialect.
Decision gate before coding: choose whether this pipeline-context surface is an internal registration helper or a public guard-rails API. The earlier milestones do not depend on that choice.

7. Rebuild the test matrix around the new policy dimensions

Update cuda_pathfinder/tests/test_compatibility_guard_rails.py to split tests into:
- same-component exact-match failures,
- independent-artifact allowance,
- driver-policy decisions,
- forward-compatibility override behavior,
- pipeline-context decisions.
Add descriptor/invariant tests for authored component metadata.
Preserve the existing process-wide routing tests in cuda_pathfinder/cuda/pathfinder/_process_wide_compatibility_guard_rails.py while expanding env-var coverage.

Important sequencing

Milestones 1 through 3 are the foundation and deliver the immediate behavioral change the issue comment asks for: local CTK coherence instead of one global CTK lockstep.
Milestones 4 and 5 together cover the CUDA_PATHFINDER_DRIVER_COMPATIBILITY comment fully.
Milestone 6 is where the model becomes truly pipeline-aware; until then, the new component-local policy already allows more than today without pretending to know pipeline semantics it cannot yet observe.

Key files to expect in the implementation

cuda_pathfinder/cuda/pathfinder/_compatibility_guard_rails.py
cuda_pathfinder/cuda/pathfinder/_process_wide_compatibility_guard_rails.py
cuda_pathfinder/cuda/pathfinder/_dynamic_libs/descriptor_catalog.py
cuda_pathfinder/cuda/pathfinder/_headers/header_descriptor_catalog.py
cuda_pathfinder/cuda/pathfinder/_utils/driver_info.py
cuda_pathfinder/tests/test_compatibility_guard_rails.py

Note to self mostly:

Coincidentally I saw .ci/pytorch/smoke_test/smoke_test.py in pytorch/pytorch#194056:

https://github.com/pytorch/pytorch/pull/194056/changes#diff-c629948d2bb1c874955838190a9cd19f154709c0a36e071e5da91abb457d9b05

At first glance, it seems there is conceptual overlap with the "guard rails" idea.

It maybe worth pointing the agent for the guard rails work to pytorch, to see if we can learn something.

[FEA]: Explore pathfinder "scoped search" feature #1038

Description

Problem to solve

Potential approach (with low maintenance as a goal in mind)

Activity

rwgk commented on Oct 10, 2025

rwgk commented on Apr 10, 2026

First Cut

Important Findings

MVP Shape

Questions

rwgk commented on Apr 28, 2026

Issue #1038 and PR #1977: where I currently stand (2026-04-28)

High-level summary

How I currently decompose issue 1038

1. Pivot library / scoped search

2. Compatibility aspects

Where PR #1595 fits in

Why I decided to focus on catching incompatibilities

Bird's-eye view of the PR 1977 design

Why a process-wide singleton?

Side note

What the compatibility checks currently do in PR 1977

1. Every resolved item is normalized into metadata

2. CTK metadata is derived in two main ways

3. Driver version is queried separately

4. The first CTK-backed item becomes the anchor

5. v1 currently requires an exact CTK major.minor match

6. Driver libraries are treated specially

7. The current driver guard rail is intentionally coarse

8. Optional constraints can also be imposed

9. Metadata insufficiency is handled separately from true incompatibility

10. Noteworthy limitation: guard rails only see what flows through pathfinder

What PR 1977 does not do

Why I still think this is the right first step

1. It already catches real, confusing failures

2. It gives us a simpler architecture to iterate on

Current real-life testing beyond mocks

Rollout strategy

Bottom line

rwgk commented on Apr 29, 2026

Issue 1038 / PR 1977 Minor Version Compatibility Research

Docs

What NVIDIA says

Key implications

What this means for guard rails

rwgk commented on Apr 29, 2026

What NVIDIA says

1. The compatibility model: three distinct mechanisms

2. Minor-version compatibility: driver ranges and caveats

3. Forward compatibility: a separate, narrower mechanism

4. NVRTC

5. libNVVM and NVVM IR

6. nvJitLink

7. PTX Compiler API

8. Background reference

rwgk commented on Apr 30, 2026

Granular CTK and Pipeline Compatibility Checks

Separate Compatibility Questions

Dynamic-Link Components

Independent CTK Artifacts

Driver Compatibility

Compiler and JIT Pipelines

Proposed Decision Tree

Pipeline Decision Tree

Practical First Implementation

Summary

rwgk commented on Apr 30, 2026

CUDA_PATHFINDER_DRIVER_COMPATIBILITY shape

Goal

Recommended shape

Why this should not be another mode

Scope of the override

Linux-only behavior

Error-message behavior

Recommendation

Future extensions

rwgk commented on Apr 30, 2026

warning_only shape

Core idea

`CUDA_PATHFINDER_DRIVER_COMPATIBILITY` shape

`warning_only` shape

Why this belongs in `_COMPATIBILITY_GUARD_RAILS_MODES`

Relationship to `assume_forward_compatibility`

5. Upgrade driver metadata beyond `cuDriverGetVersion()`