Skip to content

[CI failure] CI: Nightly optional-deps scheduled runs #2748

Closed
Closed
[CI failure] CI: Nightly optional-deps scheduled runs#2748
Assignees
leofangjpascucci-nv
Labels
CI/CDCI/CD infrastructureP1Medium priority - Should dobugSomething isn't workingci-workflow-healthAssigned to rolling issues managed by the ci-workflow-health.yml workflow.
Milestone
cuda.bindings 13.5.0 & 12.9.10

Description

@github-actions
github-actions
bot
opened on Sep 1, 2026

This issue tracks an unhealthy unattended CI workflow.
Subsequent failures are recorded in comments; a successful run in the
same trigger scope closes the issue automatically.

First unhealthy run

CI: Nightly optional-deps run #156, attempt 1 concluded with failure.

  • Trigger: schedule
  • Branch: main
  • Commit: b959d11
  • Completed: 2026-09-01T04:07:00Z

Created automatically by the CI workflow-health monitor.

Activity

  1. github-actions
    added
    bugSomething isn't working
    triageNeeds the team's attention
    CI/CDCI/CD infrastructure
    ci-workflow-healthAssigned to rolling issues managed by the ci-workflow-health.yml workflow.
    on Sep 1, 2026

  2. github-actions commented on Sep 2, 2026

    @github-actions
    github-actionsbot
    on Sep 2, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #157, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: b81f9d6
    • Completed: 2026-09-02T04:15:53Z

  3. jpascucci-nv commented on Sep 2, 2026

    @jpascucci-nv
    jpascucci-nv
    on Sep 2, 2026
    Contributor

    This appears to be a dupe of #2725 .

  4. jpascucci-nv
    added
    duplicateThis issue or pull request already exists
    on Sep 2, 2026

  5. github-actions commented on Sep 3, 2026

    @github-actions
    github-actionsbot
    on Sep 3, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #158, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: 53b4374
    • Completed: 2026-09-03T03:47:56Z

  6. jpascucci-nv commented on Sep 3, 2026

    @jpascucci-nv
    jpascucci-nv
    on Sep 3, 2026
    Contributor

    This is failing with a new problem: module 'numpy' has no attribute 'row_stack'
    Removing duplicate tag.
    This appears to be NVIDIA/numba-cuda#907 which some comments say should be fixed, but perhaps I don't understand the flow.

  7. jpascucci-nv
    removed
    duplicateThis issue or pull request already exists
    on Sep 3, 2026

  8. github-actions commented on Sep 4, 2026

    @github-actions
    github-actionsbot
    on Sep 4, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #159, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: fb57dbf
    • Completed: 2026-09-04T03:16:52Z

  9. github-actions commented on Sep 6, 2026

    @github-actions
    github-actionsbot
    on Sep 6, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #160, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: f84a7cf
    • Completed: 2026-09-06T01:41:56Z

  10. github-actions commented on Sep 7, 2026

    @github-actions
    github-actionsbot
    on Sep 7, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #161, attempt 1 concluded with cancelled.

    • Trigger: schedule
    • Branch: main
    • Commit: f84a7cf
    • Completed: 2026-09-07T02:34:58Z

  11. github-actions commented on Sep 8, 2026

    @github-actions
    github-actionsbot
    on Sep 8, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #162, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: f84a7cf
    • Completed: 2026-09-08T01:45:28Z

  12. github-actions commented on Sep 9, 2026

    @github-actions
    github-actionsbot
    on Sep 9, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #163, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: f84a7cf
    • Completed: 2026-09-09T01:42:58Z

  13. 24 remaining items

  14. github-actions commented on Sep 29, 2026

    @github-actions
    github-actionsbot
    on Sep 29, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #184, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: 36e4d40
    • Completed: 2026-09-29T03:56:08Z

  15. leofang commented on Sep 30, 2026

    @leofang
    leofang
    on Sep 30, 2026
    Member

    Ran a classification pass across nightly optional-deps runs #156–#184 (sampled runs #156, #158, #163, #170, #175, #180, #184). Setting aside the aarch64 standard slot regressions (each window brings a distinct one; not obviously actionable as a single class), the current recurring failures are three persistent classes:

    1. numba-cuda ↔ NumPy 2.5 (6 jobs — chronic since #156)

    All 6 numba-cuda jobs — {linux-64, linux-aarch64, win-64} × {CUDA 12.9.1, 13.4.2}, Python 3.12 — fail at collection with:

    File ".../numba_cuda/numba/cuda/np/arrayobj.py", line 6860
  overload(np.row_stack)(impl_np_vstack)
AttributeError: module 'numpy' has no attribute 'row_stack'

    np.row_stack was removed in NumPy 2.0; the currently-published numba-cuda 0.30.4 still references it, so nightly always resolves to a broken combination. Persisted across the numpy 2.5.2 → 2.5.3 pin bump around #163, so this is a numba-cuda 0.30.4 issue, not a numpy-version boundary. Tracked upstream in NVIDIA/numba-cuda#907 — the fix landed there but no released wheel picks it up yet. (Small correction to an earlier comment: this signature was already present in #156, one run before it was first flagged.)

    2. numba-cuda-mlir NVRTC cannot find <nv/target> (4 jobs — since #170)

    All 4 numba-cuda-mlir jobs — {linux-64, win-64} × {CUDA 12.9.1, 13.4.2}, Python 3.12 — see 2–3 test failures each (~4700–4820 passing):

    cuda.core._utils.cuda_utils.NVRTCError: 6: NVRTC_ERROR_COMPILATION
.../nvidia/cu13/include/cooperative_groups/details/info.h(50):
  catastrophic error: cannot open source file "nv/target"

    Failing tests: test_external_cooperative_func, test_cooperative_grid_sync_extern on all four; CUDA-13 configs also fail test_include_cuda_header (the curand_kernel.h fallback works under CUDA 12 but not CUDA 13). The same missing header shows up as nvcc-mode skips in test_nvjitlink.py. Signature points at a cuda.core.Program.compile include-path / cccl-header packaging issue, not a numba-cuda-mlir logic bug. First appearance: #170 (2026-09-15).

    3. cuda.core frozen fallback table behind driver (2 jobs — since #170)

    Both cuda-core jobs (linux-64 py3.12, win-64 py3.14) trip the same single assertion in tests/test_utils_enum_explanations_helpers.py::test_frozen_driver_table_covers_all_curesult_members:

    AssertionError: Missing frozen fallback explanations for:
  ['CUDA_ERROR_MULTICAST_RESOURCE_FULL',
   'CUDA_ERROR_INSUFFICIENT_LOADER_VERSION',
   'CUDA_ERROR_FABRIC_NOT_READY']

    Stale _FALLBACK_EXPLANATIONS — three newer CUresult values need entries. First appearance: #170 (2026-09-15).

    Summary

    Run  Date    numba-cuda(6)  numba-cuda-mlir(4)  cuda-core(2)
156  09-01    yes             no                  no
158  09-03    yes             no                  no
163  09-08    yes             no                  no
170  09-15    yes             yes                 yes
175  09-20    yes             yes                 yes
180  09-25    yes             yes                 yes
184  09-29    yes             yes                 yes

    -- Leo's bot

  16. leofang
    added a commit that references this issue on Sep 30, 2026

    ci: skip numba-cuda IPC tests and mlir test_device_record_copy

    f106b1f

  17. github-actions commented on Sep 30, 2026

    @github-actions
    github-actionsbot
    on Sep 30, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #186, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: e8d9075
    • Completed: 2026-09-30T03:18:24Z
  18. leofang
    added a commit that references this issue on Sep 30, 2026

    ci: clean up recurring nightly optional-deps failures (#2968)

    4b9dcbd

  19. leofang commented on Sep 30, 2026

    @leofang
    leofang
    on Sep 30, 2026
    Member

    With #2968 and #2974 I expect to see this issue closed by tonight's nightly run. Let's keep an eye on it!

  20. github-actions commented on Oct 1, 2026

    @github-actions
    github-actionsbot
    on Oct 1, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #191, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: 52e162b
    • Completed: 2026-10-01T03:20:50Z

  21. jpascucci-nv commented on Oct 1, 2026

    @jpascucci-nv
    jpascucci-nv
    on Oct 1, 2026
    Contributor

    @leofang It failed again. All the failing targets failed on the below. Should I open a separate issue, or just track it here?

    =================================== FAILURES ===================================
    _______________ CUDAAndCPUCachingTest.test_cpu_and_cuda_targets ________________

    self = <numba.cuda.tests.cudapy.test_caching.CUDAAndCPUCachingTest testMethod=test_cpu_and_cuda_targets>

    @skip_on_standalone_numba_cuda
def test_cpu_and_cuda_targets(self):
    # The same function jitted for CPU and CUDA targets should maintain
    # separate caches for each target.
    self.check_pycache(0)
    mod = self.import_module()
    self.check_pycache(0)

    f_cpu = mod.assign_cpu
    f_cuda = mod.assign_cuda
    self.assertPreciseEqual(f_cpu(5), 5)
    self.check_pycache(2)  # 1 index, 1 data
    self.assertPreciseEqual(f_cuda(5), 5)

      self.check_pycache(3)  # 1 index, 2 data

        ^^^^^^^^^^^^^^^^^^^^^

    /opt/hostedtoolcache/Python/3.12.14/x64/lib/python3.12/site-packages/numba_cuda/numba/cuda/tests/cudapy/test_caching.py:681:

    /opt/hostedtoolcache/Python/3.12.14/x64/lib/python3.12/site-packages/numba_cuda/numba/cuda/tests/cudapy/test_caching.py:93: in check_pycache
    self.assertEqual(len(c), n, c)
    E AssertionError: 2 != 3 : ['cuda_and_cpu_caching_test_fodder.target_shared_assign-20.py312.1.nbc', 'cuda_and_cpu_caching_test_fodder.target_shared_assign-20.py312.nbi']
    ________________ CUDAAndCPUCachingTest.test_cpu_and_cuda_reuse _________________

    self = <numba.cuda.tests.cudapy.test_caching.CUDAAndCPUCachingTest testMethod=test_cpu_and_cuda_reuse>

    @skip_on_standalone_numba_cuda
def test_cpu_and_cuda_reuse(self):
    # Existing cache files for the CPU and CUDA targets are reused.
    mod = self.import_module()
    mod.assign_cpu(5)
    mod.assign_cpu(5.5)
    mod.assign_cuda(5)
    mod.assign_cuda(5.5)

    mtimes = self.get_cache_mtimes()

    # Two signatures compiled
    self.check_hits(mod.assign_cpu.func, 0, 2)
    self.check_hits(mod.assign_cuda.func, 0, 2)

    mod2 = self.import_module()
    self.assertIsNot(mod, mod2)
    f_cpu = mod2.assign_cpu
    f_cuda = mod2.assign_cuda

    f_cpu(2)

      self.check_hits(f_cpu.func, 1, 0)

    /opt/hostedtoolcache/Python/3.12.14/x64/lib/python3.12/site-packages/numba_cuda/numba/cuda/tests/cudapy/test_caching.py:715:

    /opt/hostedtoolcache/Python/3.12.14/x64/lib/python3.12/site-packages/numba_cuda/numba/cuda/tests/cudapy/test_caching.py:160: in check_hits
    self.assertEqual(sum(st.cache_hits.values()), hits, st.cache_hits)
    E AssertionError: 0 != 1 : Counter()
    =========================== short test summary info ============================
    FAILED cudapy/test_caching.py::CUDAAndCPUCachingTest::test_cpu_and_cuda_targets - AssertionError: 2 != 3 : ['cuda_and_cpu_caching_test_fodder.target_shared_assign-20.py312.1.nbc', 'cuda_and_cpu_caching_test_fodder.target_shared_assign-20.py312.nbi']
    FAILED cudapy/test_caching.py::CUDAAndCPUCachingTest::test_cpu_and_cuda_reuse - AssertionError: 0 != 1 : Counter()

  22. isVoid commented on Oct 1, 2026

    @isVoid
    isVoid
    on Oct 1, 2026
    Contributor

    This is a dependency-compatibility issue in numba-cuda 0.30.4, not a separate CUDA/platform failure.

    Numba 0.68 changed the source freshness stamp in shared .nbi cache indexes from (mtime, size) to a SHA-256 content digest. Numba-CUDA still produced the old tuple, so CPU and CUDA dispatchers rejected each other's index as stale and overwrote the same .nbc data slots. That explains both failures: the file count remains 2 instead of reaching 3, and the re-imported CPU dispatcher records no cache hit.

    The selected resolution is to declare numba-cuda 0.30.x's supported Numba range as numba>=0.60,<0.67 rather than add Numba 0.68 compatibility code in maintenance mode. Because already-published numba-cuda 0.30.4 metadata cannot be changed, this CI should constrain numba<0.67 until a numba-cuda release containing the corrected dependency metadata is available.

  23. leofang commented on Oct 1, 2026

    @leofang
    leofang
    on Oct 1, 2026
    Member

    Thanks, @isVoid, let's pin it in our nightly CI here:

    cuda-python/ci/tools/run-tests

    Lines 166 to 176 in 308e4e1

    elif [[ "${test_module}" == "nightly-numba-cuda" ]]; then
    echo "Installing pathfinder + bindings + core + test deps + numba-cuda"
    # numba-cuda's test-cuXX group deps (can't use --group for a wheel install):
    # numpy<2.5: numba-cuda 0.30.4 references np.row_stack, removed in numpy 2.5. See NVIDIA/numba-cuda#907.
    PIP_ARGS+=(
    "cuda-toolkit[curand,cublas]==${TEST_CUDA_MAJOR_MINOR}.*"
    "numba-cuda[cu${TEST_CUDA_MAJOR}]"
    "cupy-cuda${TEST_CUDA_MAJOR}x"
    "numpy<2.5"
    psutil cffi pytest-xdist pytest-benchmark filecheck ml_dtypes statistics
    )

    Our nightly CI tests "Does our ToT package break any released downstream packages?", not "Do all ToT packages, ours and downstream, work together?", so unfortunately any fix in downstream would not propagate to our nightly CI.
    See More

  24. github-actions commented on Oct 2, 2026

    @github-actions
    github-actionsbot
    on Oct 2, 2026 – with GitHub Actions
    ContributorAuthor

    Another unhealthy run

    CI: Nightly optional-deps run #193, attempt 1 concluded with failure.

    • Trigger: schedule
    • Branch: main
    • Commit: 6d6af42
    • Completed: 2026-10-02T03:57:44Z

  25. isVoid commented on Oct 2, 2026

    @isVoid
    isVoid
    on Oct 2, 2026
    Contributor

    Opened #2996 to pin the nightly-numba-cuda environment to numba<0.67, as requested.

  26. github-actions commented on Oct 3, 2026

    @github-actions
    github-actionsbot
    on Oct 3, 2026 – with GitHub Actions
    ContributorAuthor

    Recovered

    CI: Nightly optional-deps run #195, attempt 1 concluded with success.

    • Trigger: schedule
    • Branch: main
    • Commit: ab9e20f
    • Completed: 2026-10-03T03:18:50Z

    Closing this incident automatically.

  27. github-actions
    closed this as completedon Oct 3, 2026
  28. leofang
    self-assigned this
    on Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

CI/CDCI/CD infrastructureP1Medium priority - Should dobugSomething isn't workingci-workflow-healthAssigned to rolling issues managed by the ci-workflow-health.yml workflow.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions