Repository navigation
Python 3.14 crash randomly on AMD64 Debian root 3.14 buildbot: CPU bug #159764
Description
On AMD64 Debian root 3.14, on some builds, Python 3.14 does crash randomly.
For example, on build 1874, test_unicodedata failed with failed C assertion, but then passed when re-run in verbose mode
Build 1874: https://buildbot.python.org/#/builders/1742/builds/1874
test_failed_import_during_compiling (test.test_unicodedata.UnicodeMiscTest.test_failed_import_during_compiling) ... ok
test_linebreak_7643 (test.test_unicodedata.UnicodeMiscTest.test_linebreak_7643) ...
python: Python/generated_cases.c.h:8398: _PyEval_EvalFrameDefault: Assertion `oparg & 1' failed.
Fatal Python error: Aborted
Current thread 0x00007fa321491740 [python] (most recent call first):
File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/unittest/case.py", line 905 in _getAssertEqualityFunc
File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/unittest/case.py", line 924 in assertEqual
File "/root/buildarea/3.14.angelico-debian-amd64/build/Lib/test/test_unicodedata.py", line 814 in test_linebreak_7643
Many tests failed once, but passed when re-run in verbose mode. Two tests failed twice:
test.test_multiprocessing_spawn.test_processes test_collections
test_math test_statistics test_unicodedata test_unparse
test_userstring test_zipimport
2 tests failed:
test_collections test_mathPython built with ./configure --prefix '$(PWD)/target' --with-pydebug on Debian GNU/Linux 12 (bookworm) using gcc -Og: gcc (Debian 12.2.0-14) 12.2.0.
Recent builds which failed:
- https://buildbot.python.org/#/builders/1742/builds/1874 (8 re-run tests)
- https://buildbot.python.org/#/builders/1742/builds/1870 (24 re-run tests)
- https://buildbot.python.org/#/builders/1742/builds/1859 (3 re-run tests)
- https://buildbot.python.org/#/builders/1742/builds/1858 (3 re-run tests)
- https://buildbot.python.org/#/builders/1742/builds/1857 (5 re-run tests)
- https://buildbot.python.org/#/builders/1742/builds/1844 (13 re-run tests)
- https://buildbot.python.org/#/builders/1742/builds/1843 (9 re-run tests)
- https://buildbot.python.org/#/builders/1742/builds/1841 (8 re-run tests)
Activity
vstinner commentedon Oct 19, 2026
I tried but failed to reproduce the issue in a Debian Bookworm (12) container with:
vstinner commentedon Oct 19, 2026
I tried other commits, but I still failed to reproduce the issue:
d26fe798c7feac115067ad0e18177f1ef0a4c9b39cf9e3859dd98f1c3d127afd0b06465857505809I built Python twice at each commit and I failed to reproduce the issue.
Example of script:
vstinner commentedon Oct 19, 2026
Build 1876 is quite bad: https://buildbot.python.org/#/builders/1742/builds/1876
Examples of bugs:
and
and
NekoAsakura commentedon Oct 20, 2026
I can't reproduce it either. Pinging @Rosuav
Rosuav commentedon Oct 20, 2026
Hmm. I could simply reboot the VM and see if that clears it up, but before I do, is there any other sort of diagnosis that would be helpful?
vstinner commentedon Oct 20, 2026
@Rosuav:
So far, I don't have any clue about the root cause. I already saw such bug on your buildbot, maybe a few months ago (I didn't take any notes about it). So I'm not sure that a reboot is enough to work around the issue. The issue might come back after the reboot.
Can you get the version of the
gccpackage usingdpkg -s gcc|grep ^Version? Also, can you upgrade Debian (sudo apt update && sudo apt upgrade) before rebooting, to check if a Debian upgrade is enough to fix the issue?As I asked you by email, if it's possible, I would be interested to get a SSH access to this buildbot (virtual) machine to debug the issue.
If it's not possible, I'm curious to know if you're able to reproduce the issue on another Debian 12 machine?
Rosuav commentedon Oct 20, 2026
Version: 4:12.2.0-3
Though notably, the
gcc-12package has a version of 12.2.0-14, and the updates that I'm about to run will make that12.2.0-14+deb12u1.Hmm, if this was recent, I didn't get that email. But yeah, that should be an option. I can create you an account on the VM, and then I'll give you a dedicated SSH port number for the buildbot. What username would you like?
vstinner commentedon Oct 20, 2026
I sent you an email today to the email address of your GitHub profile. I would like to use the username
vstinner. You can get my SSH public from GitHub. You can send me the machine hostname or IP address by email (or here if it's not an issue to make it public).Rosuav commentedon Oct 20, 2026
We got in touch via email and credentials have been organized. (Also, while that was happening, I ran updates on the buildbot, although nothing particularly obvious went through.)
vstinner commentedon Oct 20, 2026
Thanks to @Rosuav, I got access to the Debian 12 machine.
I built Python in 3 flavors:
gcc -Og(./configure --with-pydebug)gcc -O3(./configure)gcc -O0 -ggdb(./configure --with-pydebug 'CFLAGS=-O0 -ggdb')So far, I only reproduced Python crashes on the "pydebug" build (
gcc -Og). The failure rate is quite low. It's so low that I failed to compute it yet. I'm running stress tests in a loop and I only get a crash after 15 to 30 minutes (coarse estimation). One time, I got lucky and I got a crash after 29 runs, and then after 13 runs. But after than, I couldn't reproduce the crash for like 30 minutes.I'm testing the same Python binary with a stress test which use the same random seed for all test worker processes. I would expect all processes to crash or all processes to pass the test, but no, instead I get random crashes. So it seems like it's a bug with the machine rather than a bug in Python.
I reproduced crashes in test_struct and test_buffer.
I like testing test_struct since the _struct extension is simple C code (ex: no syscall) and the test is quick (less than a second to run).
There are different "crashes":
python: Python/generated_cases.c.h:8353: _PyEval_EvalFrameDefault: Assertion 'oparg & 1' failed.AssertionError: 255 != 255onself.assertEqual(x, retrieved)(not a "crash", just an exception)python: ../Python/generated_cases.c.h:4546: _PyEval_EvalFrameDefault: Assertion 'oparg == 1' failed.python: ../Python/generated_cases.c.h:8984: _PyEval_EvalFrameDefault: Assertion 'WITHIN_STACK_BOUNDS()' failed.Note: time to time, I run
./python -m test -j0 test_codecmaps_cn test_http_cookies test_math test_ntpath test_statistics test_tokenize test_unicodedata test_unparse test_userstring test_struct test_ast test_buffer test_gzip test_logging test_codecsto try to reproduce crashes.Rosuav commentedon Oct 20, 2026
Ah! Then I shall attempt to build in debug mode on the host machine. Using
./python -m test -j0 -F test_structas a stress test. Since it's able to run across ALL cores (rather than being capped at 2 inside the VM), this should mean that, if there's a hardware issue with one of the CPU cores, it's going to hit it.My understanding of
python -m test --helpis that-F, since it implies--failfast, should halt the entire test if this crash occurs. Correct? If so, I will just leave it running here for a while and check back later. It's up to three minutes and 650ish tests run.EDIT: No issues on the host machine.
0:37:42 load avg: 49.60 mem: 639.1 MiB [8385] test_struct passedand I hit Ctrl-C.hugovk commentedon Oct 25, 2026
We've now had seven failures in a row and it's blocking 3.14 releases:
Rosuav commentedon Oct 25, 2026
I just restarted the VM and it's now showing some greens. I have no idea still what is actually causing the issues.
8 remaining items
[-]Python 3.14 crash randomly on AMD64 Debian root 3.14 buildbot (VM issue or hardware issue?)[/-][+]Python 3.14 crash randomly on AMD64 Debian root 3.14 buildbot: CPU bug[/+]NekoAsakura commentedon Oct 28, 2026
Uh oh. @Rosuav could you try pinning the VM to E-cores and see if the crashes stop? Hope it's just one core tricking 🎃.
Rosuav commentedon Oct 28, 2026
Hmm. I tried pinning the VBoxHeadless process to cores 25 and 26 (
taskset -p 0x6000000 5344where 5344 happens to be its current pid) but I'm not sure that it's respecting that. I set the VM to busywait for a while, and CPU 9 was the one that got busy. Changed the affinity mask, did it again; still CPU 9 (one of the highest performing cores) was the busy one. :(NekoAsakura commentedon Oct 28, 2026
You might need
-aas vCPUs have their own threads.Rosuav commentedon Oct 28, 2026
Oh! Thank you. (Sorry, I probably come across as a pretty ignorant sysadmin, but I have never before needed to concern myself with which CPU core does what - which is broadly a good thing, I've never had a CPU with this sort of bug...) I was also off by one in my CPU numbering so I've now successfully set the affinity where I want it.
The build is now running and I'll check back in a bit. Two green lines on the CPU graph.
Rosuav commentedon Oct 28, 2026
Total duration: 2 hour 8 min. Total tests: run=283,011 skipped=14,094. Result: INTERRUPTED. I'm going to call this good enough, and let a couple of real tests go through.
hugovk commentedon Oct 29, 2026
btw #142973 could be an early signal of this same issue?
test_sampling_profiler#142973 as a duplicate of this issueNekoAsakura commentedon Nov 2, 2026
Looks like it really was just one core. 106 builds since the pin without a crash, and the warnings are mostly #160986 (with open PR #161073), so closing.
Rosuav commentedon Nov 6, 2026
As of today, the hardware has been replaced, and I've removed the CPU core pinning. If there are any further issues, I definitely need to know about them, but hopefully for now, the only effect is that we're back to full speed.
vstinner commentedon Nov 6, 2026
Replacement was quick! Did Intel send you a new CPU?
Rosuav commentedon Nov 6, 2026
I spoke to the local supplier I got it from, they were extremely helpful and got the replacement from Intel prior to having my faulty one. So I spent the past couple weeks in communication, but only had about two hours of actual downtime for the swap. It'd been on my "deal with this eventually" list ever since I found out about the problem, but until the work in this thread, it hadn't been urgent.
On the plus side, I am now able to remove the thermal throttling (where it had been set to cap 15° lower than the usual), as the system appears fully stable on all cores. I tested the buildbot (running
/python -m test -j2 -F test_struct) pinned on each CPU core and had no issues.vstinner commentedon Nov 6, 2026
Oh nice, you made an exhaustive test :-) It's great that you managed to replace the faulty CPU. It's very rare for me to hit real hardware issue in the wild. It was an interesting adventure for me to investigate it :-)
Rosuav commentedon Nov 6, 2026
Yeah, it's been a unique experience for me too. I appreciate everyone's patience with the various outages while we figured this out, and especially the deep help from everyone in this thread. @vstinner rocks. Just saying.