Repository navigation
Description
Crash report
I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests. With daemon threads, the process sometimes crashed at exit. The cause looks like a gap left after the fix for #124878 (#130649): a thread that is just starting can bind its thread state after finalization has already cleared it. An AI assistant helped with the analysis.
What happens
This script starts daemon threads that keep starting more daemon threads, and exits after 20 ms:
import threading, time
def work():
while True:
x = [bytes(100) for _ in range(300)]
t = threading.Thread(target=lambda: [bytes(64) for _ in range(100)], daemon=True)
t.start()
for _ in range(8):
threading.Thread(target=work, daemon=True).start()
time.sleep(0.02)I ran it in a loop of fresh processes on main at e8c0584 (2026-11-10), release builds, Linux arm64. A probe (a 3 ms pause in thread_run() at every 16th thread start, in the fold) makes the crash frequent:
| exits | crashed (SIGSEGV) | |
|---|---|---|
| free-threaded build | 10,752 | 1 |
| free-threaded build with the probe | 12,845 | 198 |
| default (GIL) build | 16,068 | 0 |
On macOS I saw no crash (0 of 13,034 exits). 3.14 and 3.15 have the same code; I tested main only.
Finalization crashes at a decref of an object that belongs to another thread (backtrace trimmed):
#0 find_thread_state (thread_id=281472972548224) at Python/brc.c:40
#1 _Py_brc_queue_object (ob=0x35708810160) at Python/brc.c:90
#3 dictkeys_decref (dk=0x3570c06c000, use_qsbr=true) at Objects/dictobject.c:517
#11 _PyGC_CollectNoFail (tstate=0xaaaae1580708 <_PyRuntime+295328>) at Python/gc_free_threading.c:2634
#13 _Py_Finalize (runtime=0xaaaae1538568 <_PyRuntime>) at Python/pylifecycle.c:2501
find_thread_state() looks up the owning thread in the table of threads that biased reference counting keeps. In the core file, it reaches a thread state whose memory is all zero: freed, but still in the table.
Why I think it happens
In thread_run(), a new thread checks whether it must exit (_PyThreadState_MustExit()) and then binds its thread state (_PyThreadState_Bind(), which adds it to the table). If finalization starts between the two:
- Finalization marks the thread state as shutting down and clears it. The state isn't in the table yet, so
_Py_brc_remove_thread()has nothing to remove. The interpreter drops its reference. - The thread binds the cleared state, which adds it to the table. It fails to attach, hangs in
_PyThreadState_HangThread()and drops the last reference. The state is freed but stays in the table. - Finalization's next decref of an object owned by another thread searches that part of the table and crashes.
Only daemon threads get this far; finalization waits for the others first. The probe only makes the gap between the check and the bind longer.
PyThreadState_New() has the same gap: _PyThreadState_NewBound() creates the thread state and binds it right after. With a small C extension whose own threads keep creating and deleting thread states while the process exits (in the fold), and the same kind of pause between creating and binding, a debug build aborted on the assertion in bind_tstate() in 3,450 of 3,450 exits. PyGILState_Ensure() seems safe on main: it holds an interpreter guard from PEP 788 while it creates and binds, and the extension changed to use it didn't crash (0 of 12,868 exits, release build).
As far as I can tell, both cases come from one assumption: finalization (_PyThreadState_DeleteList()) clears the thread states of the other threads as if a thread that isn't attached didn't use its thread state. A thread that is binding one does.
Related reports
- race condition in threading when interpreter finalized while daemon thread runs (thread sanitizer identified) #124878: since its fix, the new thread holds its own reference, so the state isn't freed too early. The bind after the clear remains.
- Threads started in exit handler are still running after their thread states are destroyed #104690 (2023): the same assertion in
bind_tstate(). Here a debug build with the probe trips it in 4,773 of 5,058 exits, without the probe in 0 of 2,485. - test_threading fails and crashes when rerun #108987: where the "must exit" check in
thread_run()comes from. - PyGILStateEnsure() (Probably) Crashes If Called After Finalization #124619: threads not started by Python that create a thread state only after finalization has taken the others out. A different window, which the C extension also hits.
A sketch that worked here
I don't know this code well enough to say whether this is the right place for a fix, but this sketch removed the crash in my tests. A small field in each thread state decides who goes first. The thread claims it before binding and marks it as bound afterwards (in _PyThreadState_Bind() and _PyThreadState_NewBound()); finalization claims it before clearing (in _PyThreadState_DeleteList()). If finalization is first, the thread leaves the state unbound and hangs when it tries to attach, so nothing of it is in the table. If the thread is in the middle of its bind, finalization waits for it and then clears the state. After a fork there is no waiting: the binding thread doesn't exist in the child.
| exits without the sketch | crashed or aborted | exits with the sketch | crashed or aborted | |
|---|---|---|---|---|
| the script, release build | 10,752 | 1 | 10,550 | 0 |
| the script with the probe, release build | 12,845 | 198 | 9,954 | 0 |
| the script with the probe, debug build | 5,058 | 4,773 | 3,596 | 0 |
| the C extension with the pause, debug build | 3,450 | 3,450 | 3,017 | 0 |
The C extension also hits #124619. In the last row a second sketch for that issue was applied too; I'll describe it there. With this sketch alone, the full test suite passes on Linux (free-threaded release and debug builds, default build).
Its limits:
- Without the sketch the script also hangs at exit now and then (16 of the 10,752 exits). That looks like a separate problem (
PyMutexhands a mutex directly to a thread that has waited for it more than 1 ms, and that thread may then hang), which I'm reporting separately. None of the runs with the sketch hung, but it isn't meant for that. - Finalization now waits for a thread that is in the middle of its bind. If that thread is stuck on a mutex of the table that was handed off to a hung thread, finalization waits forever; without the sketch it would get stuck on the same mutex a moment later, when it clears that thread state.
Attempts that didn't work out:
- Checking and binding while holding
HEAD_LOCK: deadlocked once in 14,335 exits. The thread heldHEAD_LOCK, which a stop-the-world needs, while it waited for a mutex of the table that had been handed to a thread the stop-the-world had already paused. - Removing the state from the table in
_PyThreadState_HangThread(): tested clean, but every hanging thread then reads its interpreter, which may already be freed for a subinterpreter. - A second "must exit" check after the bind: no crash with the probe (0 of 21,987 exits), but the cleared state stays bound (a debug build with the probe still aborts in 2,750 of 2,756 exits), and
PyThreadState_New()isn't covered.
I may well be missing context here, and I'm glad to test other approaches.
The sketch, the probe and the C extension (against e8c0584, 2026-11-10)
--- a/Include/internal/pycore_tstate.h
+++ b/Include/internal/pycore_tstate.h
@@ -39,6 +39,10 @@
// thread state during runtime finalization.
Py_ssize_t refcount;
+ // Who goes first: the thread binding this thread state, or finalization
+ // clearing it (_PyThreadState_DeleteList()). One of _Py_TSTATE_BIND_*.
+ int bind_claim;
+
// These are addresses, but we need to convert to ints to avoid UB.
uintptr_t c_stack_top;
uintptr_t c_stack_soft_limit;
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -160,6 +160,46 @@
static void tstate_mimalloc_bind(PyThreadState *);
+#define _Py_TSTATE_BIND_NONE 0
+#define _Py_TSTATE_BIND_BINDING 1 // the thread is binding it
+#define _Py_TSTATE_BIND_DONE 2 // the thread has bound it
+#define _Py_TSTATE_BIND_CLAIMED 3 // finalization took it first
+
+// Called by the thread before it binds tstate. Returns 0 if finalization has
+// already claimed the thread state: then the thread must not bind it.
+static int
+bind_claim_begin(PyThreadState *tstate)
+{
+ int expected = _Py_TSTATE_BIND_NONE;
+ return _Py_atomic_compare_exchange_int(
+ &((_PyThreadStateImpl *)tstate)->bind_claim, &expected,
+ _Py_TSTATE_BIND_BINDING);
+}
+
+static void
+bind_claim_end(PyThreadState *tstate)
+{
+ _Py_atomic_store_int(&((_PyThreadStateImpl *)tstate)->bind_claim,
+ _Py_TSTATE_BIND_DONE);
+}
+
+// Called by finalization before it clears the thread state of another thread:
+// either the thread hasn't started to bind it (and now won't), or wait until
+// its bind has finished.
+static void
+bind_claim_for_clear(PyThreadState *tstate)
+{
+ int *claim = &((_PyThreadStateImpl *)tstate)->bind_claim;
+ int expected = _Py_TSTATE_BIND_NONE;
+ if (_Py_atomic_compare_exchange_int(claim, &expected,
+ _Py_TSTATE_BIND_CLAIMED)) {
+ return;
+ }
+ while (_Py_atomic_load_int(claim) == _Py_TSTATE_BIND_BINDING) {
+ _Py_yield();
+ }
+}
+
static void
bind_tstate(PyThreadState *tstate)
{
@@ -1802,14 +1842,17 @@
_PyThreadState_NewBound(PyInterpreterState *interp, int whence)
{
PyThreadState *tstate = new_threadstate(interp, whence);
- if (tstate) {
+ if (tstate && bind_claim_begin(tstate)) {
bind_tstate(tstate);
// This makes sure there's a gilstate tstate bound
// as soon as possible.
if (gilstate_get() == NULL) {
bind_gilstate_tstate(tstate);
}
+ bind_claim_end(tstate);
}
+ // Otherwise finalization has taken (and cleared) the thread state before
+ // the bind: leave it unbound; the thread hangs when it tries to attach.
return tstate;
}
@@ -2190,6 +2233,9 @@
PyThreadState *p, *next;
for (p = list; p; p = next) {
next = p->next;
+ if (!is_after_fork) {
+ bind_claim_for_clear(p);
+ }
PyThreadState_Clear(p);
tstate_delete_common(p, /*release_gil=*/0, /*is_unlinked=*/1);
if (is_after_fork) {
@@ -2817,12 +2863,18 @@
// was called, tstate becomes a dangling pointer.
assert(_PyThreadState_CheckConsistency(tstate));
+ if (!bind_claim_begin(tstate)) {
+ // Finalization has taken the thread state before the bind: leave it
+ // unbound; the thread hangs when it tries to attach.
+ return;
+ }
bind_tstate(tstate);
// This makes sure there's a gilstate tstate bound
// as soon as possible.
if (gilstate_get() == NULL) {
bind_gilstate_tstate(tstate);
}
+ bind_claim_end(tstate);
}
#if defined(Py_GIL_DISABLED) && !defined(Py_LIMITED_API)The probe in thread_run() (not part of the sketch). The pause for the C extension is the same block in _PyThreadState_NewBound(), between new_threadstate() and bind_tstate().
--- a/Modules/_threadmodule.c
+++ b/Modules/_threadmodule.c
@@ -2,6 +2,7 @@
/* Interface to Sjoerd's portable C thread library */
#include "Python.h"
+#include <unistd.h> // usleep(), PROBE
#include "pycore_fileutils.h" // _PyFile_Flush
#include "pycore_interp.h" // _PyInterpreterState.threads.count
#include "pycore_lock.h"
@@ -382,6 +383,12 @@ thread_run(void *boot_raw)
goto exit;
}
+ { // PROBE: a pause at every 16th thread start
+ static int probe_count;
+ if ((_Py_atomic_add_int(&probe_count, 1) & 15) == 0) {
+ usleep(3000);
+ }
+ }
_PyThreadState_Bind(tstate);
PyEval_AcquireThread(tstate);
_Py_atomic_add_ssize(&tstate->interp->threads.count, 1);The C extension: 8 threads that loop over PyThreadState_New(), PyEval_RestoreThread(), 20 new bytes objects into a shared list (so that finalization decrefs objects they own), PyThreadState_Clear() and PyThreadState_DeleteCurrent(). It ran in the same kind of loop, with import natthreads, time; natthreads.start(8, [None] * 256); time.sleep(0.02) as the process.
#include <Python.h>
#include <pthread.h>
static PyInterpreterState *interp;
static PyObject *sink; /* a list owned by the main thread */
static void *
worker(void *arg)
{
for (Py_ssize_t i = (Py_ssize_t)arg;; i++) {
PyThreadState *ts = PyThreadState_New(interp);
PyEval_RestoreThread(ts);
for (int k = 0; k < 20; k++) {
PyObject *b = PyBytes_FromStringAndSize(NULL, 64);
if (b != NULL) {
PyList_SetItem(sink, (i * 20 + k) % PyList_GET_SIZE(sink), b); /* steals b */
}
}
PyThreadState_Clear(ts);
PyThreadState_DeleteCurrent();
}
return NULL;
}
static PyObject *
start(PyObject *self, PyObject *args)
{
int n;
if (!PyArg_ParseTuple(args, "iO!", &n, &PyList_Type, &sink)) {
return NULL;
}
Py_INCREF(sink);
interp = PyInterpreterState_Get();
for (int i = 0; i < n; i++) {
pthread_t t;
if (pthread_create(&t, NULL, worker, (void *)(Py_ssize_t)(i * 1000)) != 0) {
return PyErr_SetFromErrno(PyExc_OSError);
}
pthread_detach(t);
}
Py_RETURN_NONE;
}
static PyMethodDef methods[] = {{"start", start, METH_VARARGS, NULL}, {NULL, NULL, 0, NULL}};
static PyModuleDef_Slot slots[] = {{Py_mod_gil, Py_MOD_GIL_NOT_USED}, {0, NULL}};
static struct PyModuleDef module = {PyModuleDef_HEAD_INIT, "natthreads", NULL, 0, methods, slots};
PyMODINIT_FUNC PyInit_natthreads(void) { return PyModuleDef_Init(&module); }CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Output from running 'python -VV' on the command line:
Python 3.16.0a1+dev free-threading build (main, Nov 11 2026, 08:33:39) [GCC 14.2.0]
Activity