Repository navigation
EPIC: Fix the nightly CI failures #2722
Description
The nightly CI has failed on every scheduled run in the retained window — 25/25 from Aug 3 to Aug 27. The CI: Coverage workflow has been red since Aug 15, and the nightly pixi GPU job has never passed since it was added in June.
This epic collects the work to get all of them green.
The five failures
| # | Issue | Failure | Shape of the problem |
|---|---|---|---|
| 1 | #2723 | coverage html aborts |
Cython's coverage plugin cannot report on .pxd/.pxi files; the .coveragerc ignore list names 3 files by hand and two new ones landed |
| 2 | #2724 | nvidia-smi: command not found |
Container never gets GPU access — NVIDIA_VISIBLE_DEVICES is not set |
| 3a | NVIDIA/numba-cuda#907 | numba-cuda suite dies at import | numpy 2.5 removed np.row_stack; numba-cuda registers an overload for it at import time |
| 3b | #2726 + NVIDIA/numba-cuda-mlir#286 | 2 numba-cuda-mlir tests fail | A version gate turned off 12 test-deselects when only 11 should have gone |
| 3c | #2725 | released cuda-core dies at collection | cuda.bindings._test_helpers was deleted without a shim; frozen released code still imports it |
| — | #2204 | (no notifications) | Why all of the above stayed invisible; pre-existing, folded into this epic |
Activity
mdboom commentedon Aug 28, 2026
Also I would suggest #2730 so this kind of thing would become less likely in the future.
mdboom commentedon Sep 9, 2026
IMPORTANT: Now we have an over-arching problem with nightly CI not covered here: Since jobs are limited to an hour, nightly runs are now ending as "canceled" rather than "passing". We will need to separate work out to multiple jobs or some other strategy to reduce the overall runtime.