Python 3.15t segfault when importing numpy.random

@leofang

The 3.15t pathfinder "see what works" test segfaults

Running pathfinder tests with  LD:see_what_works  FH:see_what_works  BC:see_what_works
/__w/cuda-python/cuda-python/ci/tools/run-tests: line 155:  4249 Segmentation fault      (core dumped) pytest -ra -s -v --durations=0 tests/ 2>&1
      4250 Done                    | tee /tmp/pathfinder_test_log.txt
Error: Process completed with exit code 139.

https://github.com/NVIDIA/cuda-python/actions/runs/27041829125/job/79822100279?pr=2108#step:28:75

Originally posted by @leofang in #2108 (comment)

Issue 2171 report: Python 3.15t segfault reproduced locally

Human-targeted summary

I was able to reproduce the issue 2171 segfault locally by using the exact GitHub Actions Python distribution from actions/python-versions, rather than a locally built CPython 3.15t.

The failure does not currently look like a cuda_pathfinder implementation bug. The reduced reproducer is:

python -X faulthandler -c 'from numpy import random'

That segfaults with:

GitHub Actions Python: Python 3.15.0b2 free-threading build (main, Jun 3 2026, 14:43:48) [GCC 11.4.0]
NumPy wheel: numpy-2.4.6-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl
GIL disabled / free-threaded Python

The cuda_pathfinder test crash is an indirect trigger:

pytest starts.
The pytest-randomly plugin is auto-loaded.
pytest-randomly imports from numpy import random.
Importing numpy.random segfaults before pytest prints its session header.

As a practical workaround, disabling the pytest-randomly plugin avoids the crash:

python -m pytest -p no:randomly -ra -s -v --durations=0 tests/

With that workaround, the full local cuda_pathfinder test suite passed under the same reproduced environment:

973 passed, 6 skipped in 6.71s

This suggests issue 2171 should probably be tracked as an incompatibility between the GitHub Actions Python 3.15t beta build and the custom NumPy 2.4.6 cp315t wheel, exposed by pytest-randomly, rather than as a CUDA pathfinder library discovery failure.

Agent-facing full details

Original failing CI job

The original CI failure was in the PR 2108 GitHub Actions job:

https://github.com/NVIDIA/cuda-python/actions/runs/27041829125/job/79822100279?pr=2108

The manually downloaded CI job log is available locally at:

/wrk/obox/cuda-python_pr2108_run27041829125_job79822100279_linux64_py315t_cuda1330_pathfinder_segfault.log

The key CI failure was:

/__w/cuda-python/cuda-python/ci/tools/run-tests: line 155: 4249 Segmentation fault (core dumped) pytest -ra -s -v --durations=0 tests/ 2>&1
##[error]Process completed with exit code 139.

The CI environment installed only cuda_pathfinder's basic test dependency group before the crash, not test-cu13. It also installed a custom NumPy prerelease/free-threaded wheel:

numpy-2.4.6-cp315-cp315t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

Local reproduction setup

At first, local tests passed with a locally built CPython 3.15t:

Python 3.15.0b2 free-threading build (tags/v3.15.0b2:94a64bbc6ce, Jun 8 2026, 20:42:06) [GCC 13.3.0]

That local build did not reproduce the segfault, even after matching:

the basic test dependency group
the custom NumPy 2.4.6 cp315t wheel
the pytest command line
the pathfinder test strictness environment variables
LD_LIBRARY_PATH pointing to the Python lib directory
a local setup where the system CUDA toolkit was hidden via local_cuda.sh off

The decisive difference was the Python distribution itself.

Installing the exact GitHub Actions Python

The exact GitHub Actions Python asset was downloaded from:

actions/python-versions release: 3.15.0-beta.2-26891732290
asset: python-3.15.0-beta.2-linux-22.04-x64-freethreaded.tar.gz

The successful install log is:

/wrk/logs/install_actions-toolcache_python-3.15.0-beta.2-linux-22.04-x64-freethreaded_log_2026-06-09+094635.txt

Successful install summary:

Create Python 3.15.0-beta.2 folder
Copy Python binaries to hostedtoolcache folder
Create additional symlinks
Upgrading pip...
Successfully installed pip-26.1.2
Create complete file
# EXIT status=0

Installed local toolcache path:

/wrk/cpython_installs/actions-toolcache/Python/3.15.0-beta.2/x64-freethreaded

The local scripts were then adjusted to use that Python:

/wrk/build_pathfinder_py315t.sh
/wrk/test_pathfinder_py315t.sh

The reproduced build log is:

/wrk/logs/build_pathfinder_py315t_log_2026-06-09+095227.txt

Important build details:

Python 3.15.0b2 free-threading build (main, Jun 3 2026, 14:43:48) [GCC 11.4.0]
Successfully installed numpy-2.4.6
Successfully installed cuda-pathfinder-1.5.6.dev19+g4bb52eb13
Successfully installed pytest-9.0.3 pytest-mock-3.15.1 pytest-randomly-4.1.0 pytest-repeat-0.9.4

Installed packages in the reproduced environment:

cuda-pathfinder 1.5.6.dev19+g4bb52eb13
iniconfig       2.3.0
numpy           2.4.6
packaging       26.2
pip             26.1.2
pluggy          1.6.0
Pygments        2.20.0
pytest          9.0.3
pytest-mock     3.15.1
pytest-randomly 4.1.0
pytest-repeat   0.9.4

Local segfault reproduction

The reproduced failing test log is:

/wrk/logs/test_pathfinder_py315t_log_2026-06-09+095255.txt

The failure is very early:

+ pytest -ra -s -v --durations=0 tests/
/wrk/test_pathfinder_py315t.sh: line 15: 471521 Segmentation fault (core dumped) pytest -ra -s -v --durations=0 tests/
# EXIT status=139

There is no pytest session header. This means the crash happens during Python / pytest / plugin startup, before pytest begins normal collection output.

Reduced reproducer

The first minimal reduction showed:

python -VV
# ok

python -c 'import sys; print(sys._is_gil_enabled())'
# False

python -c 'import pytest; print(pytest.__version__)'
# ok, 9.0.3

python -c 'import numpy; print(numpy.__version__)'
# ok, 2.4.6

python -c 'import pytest_randomly; print(pytest_randomly.__version__)'
# segfault

pytest-randomly imports NumPy random at module import time:

/wrk/forked/cuda-python/Py3150b2Venv/lib/python3.15t/site-packages/pytest_randomly/__init__.py

Relevant code:

# numpy
try:
    from numpy import random as np_random

    have_numpy = True
except ImportError:  # pragma: no cover
    have_numpy = False

Further reduction confirms pytest-randomly is not necessary for the crash:

python -X faulthandler -c 'from numpy import random'
python -X faulthandler -c 'import numpy.random'
python -X faulthandler -c 'import numpy.random._pickle'
python -X faulthandler -c 'import numpy.random.mtrand'

These all segfaulted in the reproduced environment.

Faulthandler evidence

Representative command:

python -X faulthandler -c 'import pytest_randomly'

Representative Python stack:

Fatal Python error: Segmentation fault

<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
  File "/wrk/forked/cuda-python/Py3150b2Venv/lib/python3.15t/site-packages/numpy/random/_pickle.py", line 7 in <module>
  File "/wrk/forked/cuda-python/Py3150b2Venv/lib/python3.15t/site-packages/numpy/random/__init__.py", line 180 in <module>
  File "/wrk/forked/cuda-python/Py3150b2Venv/lib/python3.15t/site-packages/numpy/__init__.py", line 718 in __getattr__
  File "/wrk/forked/cuda-python/Py3150b2Venv/lib/python3.15t/site-packages/pytest_randomly/__init__.py", line 51 in <module>
  File "<string>", line 1 in <module>

Representative native stack components:

libpython3.15t.so.1.0
numpy/_core/_multiarray_umath.cpython-315t-x86_64-linux-gnu.so
numpy/random/bit_generator.cpython-315t-x86_64-linux-gnu.so
numpy/random/_mt19937.cpython-315t-x86_64-linux-gnu.so
numpy/random/mtrand.cpython-315t-x86_64-linux-gnu.so

Loaded extension modules at crash time:

numpy._core._multiarray_umath
numpy.linalg._umath_linalg
numpy.random._common
numpy.random.bit_generator
numpy.random._bounded_integers
numpy.random._pcg64
numpy.random._generator
numpy.random._mt19937
numpy.random._philox
numpy.random._sfc64
numpy.random.mtrand

Workaround verification

Disabling the pytest-randomly plugin avoids importing numpy.random during pytest startup:

python -m pytest -p no:randomly -ra -s -v --durations=0 tests/

This passed locally in the reproduced environment:

973 passed, 6 skipped in 6.71s

The skipped tests were expected in this local environment:

SKIPPED [3] tests/conftest.py:43: libnvcudla.so is not loadable via canary subprocess on this host.
SKIPPED [2] tests/test_find_nvidia_headers.py:127: nvidia-cutlass not available on Python 3.15 (scipy missing)
SKIPPED [1] tests/test_utils_env_vars.py:93: Windows-specific case-folding check

Current interpretation

The most likely causal chain is:

GitHub Actions Python 3.15t beta build
+ custom NumPy 2.4.6 cp315t wheel
+ import numpy.random
= native segfault

The cuda_pathfinder test suite is not required to reproduce the crash. It only encounters the issue because its test dependency group includes pytest-randomly, and pytest auto-loads installed plugins by default.

The reason local testing originally passed is that the locally built CPython 3.15t was not identical to the GitHub Actions Python distribution:

Local non-crashing Python:
Python 3.15.0b2 free-threading build (tags/v3.15.0b2:94a64bbc6ce, Jun 8 2026, 20:42:06) [GCC 13.3.0]

Actions crashing Python:
Python 3.15.0b2 free-threading build (main, Jun 3 2026, 14:43:48) [GCC 11.4.0]

Suggested next steps

For unblocking cuda-python CI, the narrow workaround is to disable pytest-randomly for Python 3.15t jobs, for example by passing:

-p no:randomly

or by setting:

PYTEST_DISABLE_PLUGIN_AUTOLOAD=1

and explicitly loading only required plugins. The -p no:randomly workaround is smaller and was verified locally.

For root cause, report the minimal reproducer to the owner of the custom NumPy cp315t wheel / NumPy prerelease build process:

python -X faulthandler -c 'from numpy import random'

with these details:

Python: GitHub Actions python-3.15.0-beta.2-linux-22.04-x64-freethreaded
Python version string: Python 3.15.0b2 free-threading build (main, Jun 3 2026, 14:43:48) [GCC 11.4.0]
GIL enabled: False
NumPy: 2.4.6 cp315t custom wheel
Crash module path: numpy.random / numpy.random.mtrand / numpy.random.bit_generator

It may also be useful to compare against:

a newer actions/python-versions 3.15t build, if available
a locally rebuilt NumPy wheel against the exact Actions Python
the same wheel under the locally built CPython 3.15t that did not crash
Python 3.15 non-free-threaded, if a comparable NumPy wheel is available

@seberg

Hi @seberg , I assigned this to you because it seems to be a pure Python vs NumPy compatibility issue.

I put everything you need to pick up from where I left off here:

/home/scratch.rgrossekunst_sw/random_stuff/cuda-python_issue2171_2026-06-09+1000/README.md

Please let me know any questions about the files there.

@leofang What priority would you give to this issue? It seems likely that others will run into the same issue, and it'll get resolved even if we just wait for new Python 3.15.0 releases.

However, would it make sense to bump to 24.04 for our CI, because that Python version works.

Sounds good to me in general, I just looked, there is a 26.04 already, although very recent. Modernizing to 24.04 seems like a good conservative move if we're having issues with 22.04. I'd be surprised though if that didn't tickle some other things that need tweaks and workarounds, but at least worth a try?

In case anyone here was curious: This was actually an UB (memcpy with null pointer) bug in Python that has been fixed. It just seems it showed up randomly across various builds.

It seems that this (or something very similar) is back? https://github.com/NVIDIA/cuda-python/actions/runs/29724266375/job/88295548096?pr=2390

@mdboom

Yesterday I noticed that the Python 3.15 and 3.15t pathfinder tests running under ci.yml passed before PR #2376 when using Python 3.15.0b3, then segfaulted (test-linux-64 only) 12/12 times across the PR and post-merge main runs under 3.15.0b4.

In the meantime @mdboom determined (PR #2392) that importing numpy.random triggers the segfault again.

FYI, if #2401 can fix it, we should not try to fix it ourselves

Yeah, probably best to try gh-2401 first (it's needed anyway), sorry I haven't looked at earlier. Both NumPy and Python fixed the previous issue, so this must be new.
I am wondering again if there is a chance of an ABI break between beta 2 (build) and beta 4 (runtime) but I can't see direct evidence (even if it seems what change); if it is ABI 2401 may not help and the build job Python versions may be the next to try bumping.

OK, I dug into it with the help of an agent and finally found that there was an ABI break in b4. The PyThreadState struct got a new field before the current_exception and Cython likes to optimize things like PyErr_Occurred() (which apparently get's called at import time or so).

I'll try to just pin to b2 for the test-time (matching cibuildwheel), but it's annoying of course, latest at the time of rc1 cuibuildwheel will soon after bump to that and we'll have to remove it again (no need for an exact pin at that point).

OK, I dug into it with the help of an agent and finally found that there was an ABI break in b4. The PyThreadState struct got a new field before the current_exception and Cython likes to optimize things like PyErr_Occurred() (which apparently get's called at import time or so).

I'll try to just pin to b2 for the test-time (matching cibuildwheel), but it's annoying of course, latest at the time of rc1 cuibuildwheel will soon after bump to that and we'll have to remove it again (no need for an exact pin at that point).

Thanks for investigating this! The segfault is blocking our CI. So numpy wheel for b4 is not available at the moment?

#2433 works around by testing with beta-2. Not ideal, but cibuildwheel is on beta 2 and downgrading setup-python to this seemed more reasonable then trying to build with beta 4.

NumPy also still builds wheels with beta 2, but that doesn't matter, this isn't a NumPy issue, it's a Python change that all Cython modules get affected by: If NumPy works, cuda.bindings will just crash next.

When the RC comes out this will all get resolved, until then the only thing I could think of is asking cibuildwheel to do a release that uses b4 so both parts are on b4.

As mentioned just to note: When cibuildwheel is updated next time, we'll have to revert gh-2433 and after that 3.15 should be stable and settled (because it'll be RC phase probably, so no more weird breaking 🤞).

Navigation Menu

Repository navigation

Python 3.15t segfault when importing numpy.random #2171

Issue 2171 report: Python 3.15t segfault reproduced locally

Human-targeted summary

Agent-facing full details

Original failing CI job

Local reproduction setup

Installing the exact GitHub Actions Python

Local segfault reproduction

Reduced reproducer

Faulthandler evidence

Workaround verification

Current interpretation

Suggested next steps

4 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Python 3.15t segfault when importing numpy.random #2171

Description

Activity

rwgk commented on Jun 9, 2026

Issue 2171 report: Python 3.15t segfault reproduced locally

Human-targeted summary

Agent-facing full details

Original failing CI job

Local reproduction setup

Installing the exact GitHub Actions Python

Local segfault reproduction

Reduced reproducer

Faulthandler evidence

Workaround verification

Current interpretation

Suggested next steps

rwgk commented on Jun 9, 2026

4 remaining items

rwgk commented on Jun 12, 2026

seberg commented on Jun 15, 2026

mdboom commented on Jul 20, 2026

rwgk commented on Jul 21, 2026

leofang commented on Jul 21, 2026

seberg commented on Jul 22, 2026

seberg commented on Jul 28, 2026

lijinf2 commented on Jul 28, 2026

seberg commented on Jul 28, 2026

seberg commented on Jul 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions