Nightly regression: 6 tests in test_graph_builder.py fail intermittently on linux-aarch64 py3.14/3.14t since #2008 #2290

Activity

1. leofang

   added this to the cuda.core v1.1.0 milestone on Jul 1, 2026
2. leofang

   added

   bugSomething isn't workingSomething isn't working

   P0High priority - Must do!High priority - Must do!

   cuda.coreEverything related to the cuda.core moduleEverything related to the cuda.core module

   blockedThis task is currently blocked by other tasksThis task is currently blocked by other tasks

   and removed

   blockedThis task is currently blocked by other tasksThis task is currently blocked by other tasks

   on Jul 1, 2026
3. leofang

   assigned

   Andy-Jost

   on Jul 1, 2026
4. Andy-Jost commented on Jul 6, 2026

   

   Andy-Jost

   on Jul 6, 2026

   Contributor

   More actions

   Root cause: stale/version-skewed cuda.core binary — not a runtime bug or test-ordering issue

   I dug into the actual failing job log (run 28418107489, job 84205333659) rather than the summarized error text, and I'm confident these 6 failures are a build-artifact version skew: the compiled cuda.core.graph._graph_builder under test was built from pre-#2008 source, while the test files and .pyx on disk are post-#2008. Every failure is a new test hitting old compiled behavior, and the reported .pyx line numbers map to the pre-#2008 file while pytest prints the post-#2008 source.

   Evidence (all six point the same way):

   • test_graph_embed_non_builder → AttributeError: 'object' object has no attribute '_building_ended' at _graph_builder.pyx:689. _building_ended exists nowhere in cuda.core: Cythonize GraphBuilder and Graph with handle-layer cleanup #2008 or current source, but at dad6a421df^ (the commit before cuda.core: Cythonize GraphBuilder and Graph with handle-layer cleanup #2008) line 689 is exactly if not child._building_ended:. In cuda.core: Cythonize GraphBuilder and Graph with handle-layer cleanup #2008, embed(self, GraphBuilder child) is Cython-typed, so object() yields TypeError at the call boundary.
   • test_graph_update_after_source_close → TypeError: int() ... not 'NoneType' at :843. Pre-cuda.core: Cythonize GraphBuilder and Graph with handle-layer cleanup #2008 line 841 is if not source._building_ended: and the old update() deref'd a None handle; cuda.core: Cythonize GraphBuilder and Graph with handle-layer cleanup #2008 checks _state == CLOSED first and :843 is inside a docstring.
   • test_graph_begin_building_twice → the traceback shows :304 raise RuntimeError("...already building.") and then descends into handle_return → CUDAError(CUDA_ERROR_ILLEGAL_STATE). A bare raise cannot call handle_return; the line table is from the old binary while pytest prints the new source.
   • test_graph_split_requires_building ("DID NOT RAISE") and test_graph_complete_after_close_forked ("Graph has not finished building" instead of "...has been closed") match the old driver-status/_building_ended logic and are impossible under the current per-instance _state machine + GB_check_open.
   • test_graph_close_is_idempotent → int(None). Old Graph.handle returned None after close; current as_py() returns CUgraphExec(0), so int(...) == 0 passes.

   pytest-randomly is a red herring. Under the failing run's own seed (--randomly-seed=4095826166, printed in the log) the test_graph_builder.py tests ran in exact source-file order, and the 6 failures are deterministic given the stale binary.

   Why intermittent: the cuda.core wheel is built with cibuildwheel + sccache + the nv-gha proxy cache and uploaded as an artifact; the test job downloads that wheel and runs repo-checkout tests. When the cache serves a stale _graph_builder object/.c, the wheel ships pre-#2008 code; a clean recompile passes — which is exactly why the /ok to test rerun of the same tree (58fd95efb4) went green. It is not GPU- or interpreter-specific; the py3.14/3.14t + aarch64 + L4 signature just reflects which nightly builds happened to hit the stale cache.

   Repro (any GPU host, any Python — forces the skew):

   git worktree add /tmp/pre2008 dad6a421df^
   cd /tmp/pre2008/cuda_core && pip install -v .
   git checkout dad6a421df -- cuda_core/tests/graph/test_graph_builder.py
   python -m pytest cuda_core/tests/graph/test_graph_builder.py -p no:randomly -v

   This reproduces all 6 with the _building_ended / int(None) / CUDA_ERROR_ILLEGAL_STATE tracebacks, deterministically.

   Implication: the fix belongs in the build/CI layer (bust the sccache/proxy cache for cuda.core, ensure Cython regenerates _graph_builder.c, and/or add a build-provenance smoke check), not in _graph_builder.pyx or the tests. In particular, relaxing the assertions (e.g. assert graph.handle in (0, None)) would mask a phantom — the current source already produces the intended errors. Subsequent nightlies have been green, consistent with the cache anomaly having cleared.

   @leofang — flagging for follow-up on the CI/build-cache side; happy to help add a provenance guard if that's useful.
5. leofang commented on Jul 6, 2026

   

   leofang

   on Jul 6, 2026

   Last edited by leofang

   MemberAuthor

   More actions

   Thanks, Andy (and Andy's agent)! 😄

   Closing. The analysis was right that the code and the test mismatched, but wrong that the code (binary wheel) was corrupted.

   • From here, we know that the cuda-core wheel was fetched from this commit on main (each commit on the main branch would go through the full CI tests, with wheels generated and cached in the GHA Artifact space)
   • From here, we know that the tests were fetched from this commit on main
   • the wheel commit was earlier (6/26) than the test commit (the one with cuda.core: Cythonize GraphBuilder and Graph with handle-layer cleanup #2008 merged to main, on 6/29), the reason we fetched wheels from there on 6/30 was because it's the only commit on main that had a green CI check (our binary fetch logic requires a healthy CI status), which is a rare event that could happen and be remedied once a notification system is built ([BUG]: Important CI failures are not team-visible #2204)
     • in particular, we do not rebuild the wheels during nightly tests, and there is no code- or binary- level corruption in the wheel cache/sccache infra as alluded to in the analysis
   • it would add too much complexity to the CI infra in trying to lock the commits in sync between both the code/binary and the repo (for tests), this is an acceptable risk for now
6. leofang

   closed this as not plannedon Jul 6, 2026