Python 3.14 crash randomly on AMD64 Debian root 3.14 buildbot

On AMD64 Debian root 3.14, on some builds, Python 3.14 does crash randomly.

For example, on build 1874, test_unicodedata failed with failed C assertion, but then passed when re-run in verbose mode

Build 1874: https://buildbot.python.org/#/builders/1742/builds/1874

test_failed_import_during_compiling (test.test_unicodedata.UnicodeMiscTest.test_failed_import_during_compiling) ... ok
test_linebreak_7643 (test.test_unicodedata.UnicodeMiscTest.test_linebreak_7643) ...

python: Python/generated_cases.c.h:8398: _PyEval_EvalFrameDefault: Assertion `oparg & 1' failed.
Fatal Python error: Aborted

Current thread 0x00007fa321491740 [python] (most recent call first):
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/unittest/case.py", line 905 in _getAssertEqualityFunc
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/unittest/case.py", line 924 in assertEqual
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/test/test_unicodedata.py", line 814 in test_linebreak_7643

Many tests failed once, but passed when re-run in verbose mode. Two tests failed twice:

    test.test_multiprocessing_spawn.test_processes test_collections
    test_math test_statistics test_unicodedata test_unparse
    test_userstring test_zipimport

2 tests failed:
    test_collections test_math

Python built with ./configure --prefix '$(PWD)/target' --with-pydebug on Debian GNU/Linux 12 (bookworm) using gcc -Og: gcc (Debian 12.2.0-14) 12.2.0.

Recent builds which failed:

https://buildbot.python.org/#/builders/1742/builds/1874 (8 re-run tests)
https://buildbot.python.org/#/builders/1742/builds/1870 (24 re-run tests)
https://buildbot.python.org/#/builders/1742/builds/1859 (3 re-run tests)
https://buildbot.python.org/#/builders/1742/builds/1858 (3 re-run tests)
https://buildbot.python.org/#/builders/1742/builds/1857 (5 re-run tests)
https://buildbot.python.org/#/builders/1742/builds/1844 (13 re-run tests)
https://buildbot.python.org/#/builders/1742/builds/1843 (9 re-run tests)
https://buildbot.python.org/#/builders/1742/builds/1841 (8 re-run tests)

I tried but failed to reproduce the issue in a Debian Bookworm (12) container with:

git clone https://github.com/python/cpython
cd cpython
git checkout 8e62b0898a28cdbc0bd80a97dbfef6036b4d5bcf  # build 1874
./configure --cache-file=../configure.cache --with-pydebug
make -j14
./python -m test -j0 test_codecmaps_cn test_http_cookies test_math test_ntpath test_statistics test_tokenize test_unicodedata test_unparse test_userstring

I tried other commits, but I still failed to reproduce the issue:

build 1844: d26fe798c7feac115067ad0e18177f1ef0a4c9b3
build 1870: 9cf9e3859dd98f1c3d127afd0b06465857505809

I built Python twice at each commit and I failed to reproduce the issue.

Example of script:

set -x -e
git clean -fdx
git checkout d26fe798c7feac115067ad0e18177f1ef0a4c9b3
./configure --cache-file=../configure.cache --with-pydebug
make -j14
./python -m test -j0 test_codecmaps_cn test_http_cookies test_math test_ntpath test_statistics test_tokenize test_unicodedata test_unparse test_userstring

Build 1876 is quite bad: https://buildbot.python.org/#/builders/1742/builds/1876

22 re-run tests:
    test.test_asyncio.test_base_events test.test_asyncio.test_sendfile
    test_ast test_buffer test_cext test_codecs test_context test_email
    test_gzip test_logging test_math test_pickle test_queue test_ssl
    test_statistics test_struct test_threading test_tokenize
    test_traceback test_unicodedata test_unparse test_zipimport

9 tests failed:
    test_ast test_email test_math test_pickle test_queue
    test_statistics test_tokenize test_traceback test_unparse

Examples of bugs:

0:02:01 load avg: 3.18 [ 71/537/3] test_traceback worker non-zero exit code (Exit code -6 (SIGABRT))
python: Python/generated_cases.c.h:9033: _PyEval_EvalFrameDefault: Assertion `_Py_IsImmortal(obj)' failed.
Fatal Python error: Aborted

Current thread 0x00007fc087003740 [python] (most recent call first):
  File "<frozen importlib._bootstrap>", line 311 in acquire
  File "<frozen importlib._bootstrap>", line 421 in __enter__
  File "<frozen importlib._bootstrap>", line 1368 in _find_and_load
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/ntpath.py", line 559 in <module>
  (...)

and

(...)
test_ints (test.test_pickle.PyPicklerTests.test_ints) ... ok
test_large_pickles (test.test_pickle.PyPicklerTests.test_large_pickles) ... ok
test_list_chunking (test.test_pickle.PyPicklerTests.test_list_chunking) ... python: Python/generated_cases.c.h:11686: _PyEval_EvalFrameDefault: Assertion `oparg >= 2' failed.
Fatal Python error: Aborted

Current thread 0x00007fb63f6a3740 [python] (most recent call first):
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/pickle.py", line 831 in save_long
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/pickle.py", line 572 in save
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/pickle.py", line 1023 in _batch_appends
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/pickle.py", line 1009 in save_list
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/pickle.py", line 572 in save
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/pickle.py", line 498 in dump
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/test/test_pickle.py", line 89 in dumps
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/test/pickletester.py", line 3403 in test_list_chunking
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/unittest/case.py", line 615 in _callTestMethod
  (...)

and

0:07:10 load avg: 3.71 [226/537/8] test_gzip worker non-zero exit code (Exit code -6 (SIGABRT)) -- running (1): test_statistics (30.1 sec)
python: Python/generated_cases.c.h:9130: _PyEval_EvalFrameDefault: Assertion `!PyStackRef_IsNull(GETLOCAL(oparg))' failed.
Fatal Python error: Aborted

Current thread 0x00007f475612d740 [python] (most recent call first):
  File "<frozen importlib._bootstrap_external>", line 133 in _path_join
  File "<frozen importlib._bootstrap_external>", line 1397 in find_spec
  File "<frozen importlib._bootstrap_external>", line 1270 in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1296 in find_spec
  File "<frozen importlib._bootstrap>", line 1267 in _find_spec
  File "<frozen importlib._bootstrap>", line 1333 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1371 in _find_and_load
  File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/ssl.py", line 263 in <module>
  (...)

@Rosuav

I can't reproduce it either. Pinging @Rosuav

Hmm. I could simply reboot the VM and see if that clears it up, but before I do, is there any other sort of diagnosis that would be helpful?

@Rosuav

@Rosuav:

Hmm. I could simply reboot the VM and see if that clears it up, but before I do, is there any other sort of diagnosis that would be helpful?

So far, I don't have any clue about the root cause. I already saw such bug on your buildbot, maybe a few months ago (I didn't take any notes about it). So I'm not sure that a reboot is enough to work around the issue. The issue might come back after the reboot.

Can you get the version of the gcc package using dpkg -s gcc|grep ^Version? Also, can you upgrade Debian (sudo apt update && sudo apt upgrade) before rebooting, to check if a Debian upgrade is enough to fix the issue?

As I asked you by email, if it's possible, I would be interested to get a SSH access to this buildbot (virtual) machine to debug the issue.

If it's not possible, I'm curious to know if you're able to reproduce the issue on another Debian 12 machine?

7. Can you get the version of the gcc package using dpkg -s gcc|grep ^Version?

Version: 4:12.2.0-3

Though notably, the gcc-12 package has a version of 12.2.0-14, and the updates that I'm about to run will make that 12.2.0-14+deb12u1.

7. As I asked you by email, if it's possible, I would be interested to get a SSH access to this buildbot (virtual) machine to debug the issue.

Hmm, if this was recent, I didn't get that email. But yeah, that should be an option. I can create you an account on the VM, and then I'll give you a dedicated SSH port number for the buildbot. What username would you like?

Hmm, if this was recent, I didn't get that email. But yeah, that should be an option. I can create you an account on the VM, and then I'll give you a dedicated SSH port number for the buildbot. What username would you like?

I sent you an email today to the email address of your GitHub profile. I would like to use the username vstinner. You can get my SSH public from GitHub. You can send me the machine hostname or IP address by email (or here if it's not an issue to make it public).

We got in touch via email and credentials have been organized. (Also, while that was happening, I ran updates on the buildbot, although nothing particularly obvious went through.)

@Rosuav

Thanks to @Rosuav, I got access to the Debian 12 machine.

I built Python in 3 flavors:

pydebug: gcc -Og (./configure --with-pydebug)
release: gcc -O3 (./configure)tt'
noopt: gcc -O0 -ggdb (./configure --with-pydebug 'CFLAGS=-O0 -ggdb')

So far, I only reproduced Python crashes on the "pydebug" build (gcc -Og). The failure rate is quite low. It's so low that I failed to compute it yet. I'm running stress tests in a loop and I only get a crash after 15 to 30 minutes (coarse estimation). One time, I got lucky and I got a crash after 29 runs, and then after 13 runs. But after than, I couldn't reproduce the crash for like 30 minutes.

I'm testing the same Python binary with a stress test which use the same random seed for all test worker processes. I would expect all processes to crash or all processes to pass the test, but no, instead I get random crashes. So it seems like it's a bug with the machine rather than a bug in Python.

I reproduced crashes in test_struct and test_buffer.

I like testing test_struct since the _struct extension is simple C code (ex: no syscall) and the test is quick (less than a second to run).

There are different "crashes":

python: Python/generated_cases.c.h:8353: _PyEval_EvalFrameDefault: Assertion 'oparg & 1' failed.
regular Python exception: AssertionError: 255 != 255 on self.assertEqual(x, retrieved) (not a "crash", just an exception)
python: ../Python/generated_cases.c.h:4546: _PyEval_EvalFrameDefault: Assertion 'oparg == 1' failed.
python: ../Python/generated_cases.c.h:8984: _PyEval_EvalFrameDefault: Assertion 'WITHIN_STACK_BOUNDS()' failed.

Note: time to time, I run ./python -m test -j0 test_codecmaps_cn test_http_cookies test_math test_ntpath test_statistics test_tokenize test_unicodedata test_unparse test_userstring test_struct test_ast test_buffer test_gzip test_logging test_codecs to try to reproduce crashes.

11. So far, I only reproduced Python crashes on the "pydebug" build (gcc -Og).

Ah! Then I shall attempt to build in debug mode on the host machine. Using ./python -m test -j0 -F test_struct as a stress test. Since it's able to run across ALL cores (rather than being capped at 2 inside the VM), this should mean that, if there's a hardware issue with one of the CPU cores, it's going to hit it.

My understanding of python -m test --help is that -F, since it implies --failfast, should halt the entire test if this crash occurs. Correct? If so, I will just leave it running here for a while and check back later. It's up to three minutes and 650ish tests run.

EDIT: No issues on the host machine. 0:37:42 load avg: 49.60 mem: 639.1 MiB [8385] test_struct passed and I hit Ctrl-C.

We've now had seven failures in a row and it's blocking 3.14 releases:

I just restarted the VM and it's now showing some greens. I have no idea still what is actually causing the issues.

And back to green (for now :)

https://buildbot.python.org/#/builders/1742/builds/1939

I have no idea still what is actually causing the issues.

I suspect a hardware CPU or memory bug, but so far I failed to prove it since the bug is really hard to reproduce.

Navigation Menu

Repository navigation

Python 3.14 crash randomly on AMD64 Debian root 3.14 buildbot #159764

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Python 3.14 crash randomly on AMD64 Debian root 3.14 buildbot #159764

Description

Activity

vstinner commented on Oct 19, 2026

vstinner commented on Oct 19, 2026

vstinner commented on Oct 19, 2026

NekoAsakura commented on Oct 20, 2026

Rosuav commented on Oct 20, 2026

vstinner commented on Oct 20, 2026

Rosuav commented on Oct 20, 2026

vstinner commented on Oct 20, 2026

Rosuav commented on Oct 20, 2026

vstinner commented on Oct 20, 2026

Rosuav commented on Oct 20, 2026

hugovk commented on Oct 25, 2026

Rosuav commented on Oct 25, 2026

hugovk commented on Oct 25, 2026

vstinner commented on Oct 25, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions