You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Use pytest-run-parallel to test thread safety #2114
I suspect this would need a non-trivial amount of test refactoring to avoid teardowns that interact with the CUDA driver in a stateful way? There's so much state in the CUDA driver that it's hard to imagine doing this in an automated way safely.
Maybe, maybe not. @seberg has invested a lot of efforts in CuPy and we're generally happy with us gaining more thread safety, following community standard, and having a CUDA project reaching FT-readiness. I am hopeful.
Yeah, I would propose to try it, after the CuPy release maybe I can have a look as well. CuPy had fun caches, the memory manager, and an awful lot of legacy unittest-style tests that were painful... I would think this is all much easier for cuda-python.
I am not sure what might be problematic, e.g. graph capture related tests, seems like one that one may need to just mark as thread-unsafe in a first iteration. (In CuPy they are largely fine, but not fully e.g. because module loading/unloading in multiple threads can be problematic.)
EDIT: I started to have a bit of a look.
I suspect this would need a non-trivial amount of test refactoring to avoid teardowns that interact with the CUDA driver in a stateful way?
Yeah, I now see what you mean. cuda-pathfinder tests seem fine with a few small thread-unsafe markers. But cuda-bindings/cuda-core would skip almost all tests without larger re-factor since it's fixtures need to run per-thread (e.g. due to cuInit()).
It may work to convert these to normal decorators, but without that the pytest-run-parallel run wouldn't add much.
This is implemented now running on the FT, CI jobs. A follow-up here may be to see if we can run this with TSAN to find more things and also to make it easier for downstream to use TSAN.
If we worry about missing with nogil: leading to possible deadlocks, then it could also make sense to expand it to a non-FT job.
Activity
kkraus14 commentedon May 20, 2026
I suspect this would need a non-trivial amount of test refactoring to avoid teardowns that interact with the CUDA driver in a stateful way? There's so much state in the CUDA driver that it's hard to imagine doing this in an automated way safely.
leofang commentedon May 20, 2026
Maybe, maybe not. @seberg has invested a lot of efforts in CuPy and we're generally happy with us gaining more thread safety, following community standard, and having a CUDA project reaching FT-readiness. I am hopeful.
seberg commentedon May 20, 2026
Yeah, I would propose to try it, after the CuPy release maybe I can have a look as well. CuPy had fun caches, the memory manager, and an awful lot of legacy unittest-style tests that were painful... I would think this is all much easier for cuda-python.
I am not sure what might be problematic, e.g. graph capture related tests, seems like one that one may need to just mark as thread-unsafe in a first iteration. (In CuPy they are largely fine, but not fully e.g. because module loading/unloading in multiple threads can be problematic.)
EDIT: I started to have a bit of a look.
Yeah, I now see what you mean.
cuda-pathfindertests seem fine with a few small thread-unsafe markers. Butcuda-bindings/cuda-corewould skip almost all tests without larger re-factor since it's fixtures need to run per-thread (e.g. due tocuInit()).It may work to convert these to normal decorators, but without that the
pytest-run-parallelrun wouldn't add much.seberg commentedon Sep 2, 2026
This is implemented now running on the FT, CI jobs. A follow-up here may be to see if we can run this with TSAN to find more things and also to make it easier for downstream to use TSAN.
If we worry about missing
with nogil:leading to possible deadlocks, then it could also make sense to expand it to a non-FT job.