Skip to content

pixi.lock can silently drift from ci/versions.yml's declared CUDA toolchain version #2770

Closed as duplicate

Description

@juenglin

Description

While relocking cuda_bindings/pixi.lock for an unrelated Cython version bump, we discovered that the default pixi environment (which has an unconstrained cuda-version = "*" per package.host-dependencies) had resolved to CUDA 12.9 packages (cuda-cudart, cuda-nvvm, libcufile, libnvjitlink, etc.) even though ci/versions.yml has declared cuda.build.version: "13.3.0" since 2026-05-26.

The lock was last regenerated on 2026-08-09 (#2532), three months after that version bump, and still locked to 12.9 -- most likely because full CUDA 13.3 conda-forge builds weren't yet available at relock time. Running pixi update cython (completely unrelated to CUDA versions) incidentally triggered a full re-solve of the default solve-group and picked up CUDA 13.3, producing an ~800-line unrelated diff bundled with what should have been a small, targeted lockfile change.

Why this matters

  • ci/tools/check_pixi_cuda_version.py (run in pre-commit) only validates the cuda-version text in pixi.toml against ci/versions.yml; it does not check what versions the corresponding pixi.lock actually resolved. This class of drift is invisible to both CI and pre-commit until someone runs pixi update/pixi lock for an unrelated reason and notices an unexpectedly large diff.
  • Any future targeted lockfile update (e.g. bumping a single test dependency) risks silently smuggling in a large, unrelated CUDA-toolchain bump for the unconstrained default environment. Conversely, a contributor who notices this and tries to avoid the noise by hand-editing the lock risks leaving it stale even longer.

Request

Please design and implement a robust, long-term solution that prevents pixi.lock from drifting away from ci/versions.yml's declared CUDA toolchain version -- or at minimum makes such drift immediately visible instead of silently accumulating. Possible directions (not prescriptive):

  • Extend check_pixi_cuda_version.py (or add a new check) to also parse the relevant pixi.lock entries for the unconstrained default environment and assert they match ci/versions.yml's build.version.
  • Add a scheduled CI job that relocks (or dry-run checks staleness) and fails/opens an issue if the lock has drifted from ci/versions.yml.
  • Reconsider whether the default environment's cuda-version = "*" should instead be pinned to explicitly track ci/versions.yml's build.version, removing the ambiguity at the source.

Where this was found

  • cuda_bindings/pixi.lock and cuda_core/pixi.lock both share the package.host-dependencies: cuda-version = "*" pattern for the unconstrained default environment and are susceptible.
  • cuda_pathfinder/pixi.lock does not have this pattern and was unaffected.

Activity

  1. added
    CI/CDCI/CD infrastructure
    packagingAnything related to wheels or Conda packages
    dependenciesPull requests that update a dependency file
    on Sep 4, 2026
  2. juenglin commented on Sep 4, 2026

    @juenglin
    ContributorAuthor

    Concrete build-failure risk (not just a stale-version concern): cuda_bindings/cuda/bindings/cydriver.pxd declares driver API symbols via cdef extern from 'cuda.h':, which requires the real system header at compile time rather than self-contained stubs. The checked-in generated bindings target CTK 13.3.0, and cuda_core already uses CUDA 13.1+-only driver symbols (e.g. cuDevSmResourceSplitByCount, CU_DEV_RESOURCE_TYPE_WORKQUEUE). If the unconstrained default pixi environment resolves an older CTK (as it did here, 12.9), a from-source build in that environment can fail outright on undefined identifiers for the newer symbols, rather than merely testing against a slightly stale toolkit.

    Official CI is unaffected -- ci-pixi-source-test.yml always builds via the explicitly-pinned cu12/cu13 environments, never default -- so this is scoped to local/dev use of the bare default environment. Still worth folding into whichever fix lands here.

  3. jpascucci-nv commented on Sep 8, 2026

    @jpascucci-nv
    Contributor

    I'm implementing #2298 which, if integrated, should automatically update the pixi files once a month. https://github.com/NVIDIA/cuda-python/pull/2780/changes

  4. juenglin commented on Sep 8, 2026

    @juenglin
    ContributorAuthor

    Duplicate of #2298

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CI/CDCI/CD infrastructuredependenciesPull requests that update a dependency filepackagingAnything related to wheels or Conda packages

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions