Repository navigation
pixi.lock can silently drift from ci/versions.yml's declared CUDA toolchain version #2770
Description
Description
While relocking cuda_bindings/pixi.lock for an unrelated Cython version bump, we discovered that the default pixi environment (which has an unconstrained cuda-version = "*" per package.host-dependencies) had resolved to CUDA 12.9 packages (cuda-cudart, cuda-nvvm, libcufile, libnvjitlink, etc.) even though ci/versions.yml has declared cuda.build.version: "13.3.0" since 2026-05-26.
The lock was last regenerated on 2026-08-09 (#2532), three months after that version bump, and still locked to 12.9 -- most likely because full CUDA 13.3 conda-forge builds weren't yet available at relock time. Running pixi update cython (completely unrelated to CUDA versions) incidentally triggered a full re-solve of the default solve-group and picked up CUDA 13.3, producing an ~800-line unrelated diff bundled with what should have been a small, targeted lockfile change.
Why this matters
ci/tools/check_pixi_cuda_version.py(run in pre-commit) only validates thecuda-versiontext inpixi.tomlagainstci/versions.yml; it does not check what versions the correspondingpixi.lockactually resolved. This class of drift is invisible to both CI and pre-commit until someone runspixi update/pixi lockfor an unrelated reason and notices an unexpectedly large diff.- Any future targeted lockfile update (e.g. bumping a single test dependency) risks silently smuggling in a large, unrelated CUDA-toolchain bump for the unconstrained
defaultenvironment. Conversely, a contributor who notices this and tries to avoid the noise by hand-editing the lock risks leaving it stale even longer.
Request
Please design and implement a robust, long-term solution that prevents pixi.lock from drifting away from ci/versions.yml's declared CUDA toolchain version -- or at minimum makes such drift immediately visible instead of silently accumulating. Possible directions (not prescriptive):
- Extend
check_pixi_cuda_version.py(or add a new check) to also parse the relevantpixi.lockentries for the unconstraineddefaultenvironment and assert they matchci/versions.yml'sbuild.version. - Add a scheduled CI job that relocks (or dry-run checks staleness) and fails/opens an issue if the lock has drifted from
ci/versions.yml. - Reconsider whether the
defaultenvironment'scuda-version = "*"should instead be pinned to explicitly trackci/versions.yml'sbuild.version, removing the ambiguity at the source.
Where this was found
cuda_bindings/pixi.lockandcuda_core/pixi.lockboth share thepackage.host-dependencies: cuda-version = "*"pattern for the unconstraineddefaultenvironment and are susceptible.cuda_pathfinder/pixi.lockdoes not have this pattern and was unaffected.
Activity
juenglin commentedon Sep 4, 2026
Concrete build-failure risk (not just a stale-version concern):
cuda_bindings/cuda/bindings/cydriver.pxddeclares driver API symbols viacdef extern from 'cuda.h':, which requires the real system header at compile time rather than self-contained stubs. The checked-in generated bindings target CTK 13.3.0, andcuda_corealready uses CUDA 13.1+-only driver symbols (e.g.cuDevSmResourceSplitByCount,CU_DEV_RESOURCE_TYPE_WORKQUEUE). If the unconstraineddefaultpixi environment resolves an older CTK (as it did here, 12.9), a from-source build in that environment can fail outright on undefined identifiers for the newer symbols, rather than merely testing against a slightly stale toolkit.Official CI is unaffected --
ci-pixi-source-test.ymlalways builds via the explicitly-pinnedcu12/cu13environments, neverdefault-- so this is scoped to local/dev use of the baredefaultenvironment. Still worth folding into whichever fix lands here.jpascucci-nv commentedon Sep 8, 2026
I'm implementing #2298 which, if integrated, should automatically update the pixi files once a month. https://github.com/NVIDIA/cuda-python/pull/2780/changes
juenglin commentedon Sep 8, 2026
Duplicate of #2298