Skip to content

Free-threaded build: crash at exit when a starting thread binds a thread state that finalization has cleared #162109

Open

Description

@maikschulze

Crash report

I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests. With daemon threads, the process sometimes crashed at exit. The cause looks like a gap left after the fix for #124878 (#130649): a thread that is just starting can bind its thread state after finalization has already cleared it. An AI assistant helped with the analysis.

What happens

This script starts daemon threads that keep starting more daemon threads, and exits after 20 ms:

import threading, time
def work():
    while True:
        x = [bytes(100) for _ in range(300)]
        t = threading.Thread(target=lambda: [bytes(64) for _ in range(100)], daemon=True)
        t.start()
for _ in range(8):
    threading.Thread(target=work, daemon=True).start()
time.sleep(0.02)

I ran it in a loop of fresh processes on main at e8c0584 (2026-11-10), release builds, Linux arm64. A probe (a 3 ms pause in thread_run() at every 16th thread start, in the fold) makes the crash frequent:

exits crashed (SIGSEGV)
free-threaded build 10,752 1
free-threaded build with the probe 12,845 198
default (GIL) build 16,068 0

On macOS I saw no crash (0 of 13,034 exits). 3.14 and 3.15 have the same code; I tested main only.

Finalization crashes at a decref of an object that belongs to another thread (backtrace trimmed):

#0  find_thread_state (thread_id=281472972548224) at Python/brc.c:40
#1  _Py_brc_queue_object (ob=0x35708810160) at Python/brc.c:90
#3  dictkeys_decref (dk=0x3570c06c000, use_qsbr=true) at Objects/dictobject.c:517
#11 _PyGC_CollectNoFail (tstate=0xaaaae1580708 <_PyRuntime+295328>) at Python/gc_free_threading.c:2634
#13 _Py_Finalize (runtime=0xaaaae1538568 <_PyRuntime>) at Python/pylifecycle.c:2501

find_thread_state() looks up the owning thread in the table of threads that biased reference counting keeps. In the core file, it reaches a thread state whose memory is all zero: freed, but still in the table.

Why I think it happens

In thread_run(), a new thread checks whether it must exit (_PyThreadState_MustExit()) and then binds its thread state (_PyThreadState_Bind(), which adds it to the table). If finalization starts between the two:

  1. Finalization marks the thread state as shutting down and clears it. The state isn't in the table yet, so _Py_brc_remove_thread() has nothing to remove. The interpreter drops its reference.
  2. The thread binds the cleared state, which adds it to the table. It fails to attach, hangs in _PyThreadState_HangThread() and drops the last reference. The state is freed but stays in the table.
  3. Finalization's next decref of an object owned by another thread searches that part of the table and crashes.

Only daemon threads get this far; finalization waits for the others first. The probe only makes the gap between the check and the bind longer.

PyThreadState_New() has the same gap: _PyThreadState_NewBound() creates the thread state and binds it right after. With a small C extension whose own threads keep creating and deleting thread states while the process exits (in the fold), and the same kind of pause between creating and binding, a debug build aborted on the assertion in bind_tstate() in 3,450 of 3,450 exits. PyGILState_Ensure() seems safe on main: it holds an interpreter guard from PEP 788 while it creates and binds, and the extension changed to use it didn't crash (0 of 12,868 exits, release build).

As far as I can tell, both cases come from one assumption: finalization (_PyThreadState_DeleteList()) clears the thread states of the other threads as if a thread that isn't attached didn't use its thread state. A thread that is binding one does.

Related reports

A sketch that worked here

I don't know this code well enough to say whether this is the right place for a fix, but this sketch removed the crash in my tests. A small field in each thread state decides who goes first. The thread claims it before binding and marks it as bound afterwards (in _PyThreadState_Bind() and _PyThreadState_NewBound()); finalization claims it before clearing (in _PyThreadState_DeleteList()). If finalization is first, the thread leaves the state unbound and hangs when it tries to attach, so nothing of it is in the table. If the thread is in the middle of its bind, finalization waits for it and then clears the state. After a fork there is no waiting: the binding thread doesn't exist in the child.

exits without the sketch crashed or aborted exits with the sketch crashed or aborted
the script, release build 10,752 1 10,550 0
the script with the probe, release build 12,845 198 9,954 0
the script with the probe, debug build 5,058 4,773 3,596 0
the C extension with the pause, debug build 3,450 3,450 3,017 0

The C extension also hits #124619. In the last row a second sketch for that issue was applied too; I'll describe it there. With this sketch alone, the full test suite passes on Linux (free-threaded release and debug builds, default build).

Its limits:

  • Without the sketch the script also hangs at exit now and then (16 of the 10,752 exits). That looks like a separate problem (PyMutex hands a mutex directly to a thread that has waited for it more than 1 ms, and that thread may then hang), which I'm reporting separately. None of the runs with the sketch hung, but it isn't meant for that.
  • Finalization now waits for a thread that is in the middle of its bind. If that thread is stuck on a mutex of the table that was handed off to a hung thread, finalization waits forever; without the sketch it would get stuck on the same mutex a moment later, when it clears that thread state.

Attempts that didn't work out:

  • Checking and binding while holding HEAD_LOCK: deadlocked once in 14,335 exits. The thread held HEAD_LOCK, which a stop-the-world needs, while it waited for a mutex of the table that had been handed to a thread the stop-the-world had already paused.
  • Removing the state from the table in _PyThreadState_HangThread(): tested clean, but every hanging thread then reads its interpreter, which may already be freed for a subinterpreter.
  • A second "must exit" check after the bind: no crash with the probe (0 of 21,987 exits), but the cleared state stays bound (a debug build with the probe still aborts in 2,750 of 2,756 exits), and PyThreadState_New() isn't covered.

I may well be missing context here, and I'm glad to test other approaches.

The sketch, the probe and the C extension (against e8c0584, 2026-11-10)
--- a/Include/internal/pycore_tstate.h
+++ b/Include/internal/pycore_tstate.h
@@ -39,6 +39,10 @@
     // thread state during runtime finalization.
     Py_ssize_t refcount;
 
+    // Who goes first: the thread binding this thread state, or finalization
+    // clearing it (_PyThreadState_DeleteList()). One of _Py_TSTATE_BIND_*.
+    int bind_claim;
+
     // These are addresses, but we need to convert to ints to avoid UB.
     uintptr_t c_stack_top;
     uintptr_t c_stack_soft_limit;
--- a/Python/pystate.c
+++ b/Python/pystate.c
@@ -160,6 +160,46 @@
 
 static void tstate_mimalloc_bind(PyThreadState *);
 
+#define _Py_TSTATE_BIND_NONE      0
+#define _Py_TSTATE_BIND_BINDING   1   // the thread is binding it
+#define _Py_TSTATE_BIND_DONE      2   // the thread has bound it
+#define _Py_TSTATE_BIND_CLAIMED   3   // finalization took it first
+
+// Called by the thread before it binds tstate. Returns 0 if finalization has
+// already claimed the thread state: then the thread must not bind it.
+static int
+bind_claim_begin(PyThreadState *tstate)
+{
+    int expected = _Py_TSTATE_BIND_NONE;
+    return _Py_atomic_compare_exchange_int(
+        &((_PyThreadStateImpl *)tstate)->bind_claim, &expected,
+        _Py_TSTATE_BIND_BINDING);
+}
+
+static void
+bind_claim_end(PyThreadState *tstate)
+{
+    _Py_atomic_store_int(&((_PyThreadStateImpl *)tstate)->bind_claim,
+                         _Py_TSTATE_BIND_DONE);
+}
+
+// Called by finalization before it clears the thread state of another thread:
+// either the thread hasn't started to bind it (and now won't), or wait until
+// its bind has finished.
+static void
+bind_claim_for_clear(PyThreadState *tstate)
+{
+    int *claim = &((_PyThreadStateImpl *)tstate)->bind_claim;
+    int expected = _Py_TSTATE_BIND_NONE;
+    if (_Py_atomic_compare_exchange_int(claim, &expected,
+                                        _Py_TSTATE_BIND_CLAIMED)) {
+        return;
+    }
+    while (_Py_atomic_load_int(claim) == _Py_TSTATE_BIND_BINDING) {
+        _Py_yield();
+    }
+}
+
 static void
 bind_tstate(PyThreadState *tstate)
 {
@@ -1802,14 +1842,17 @@
 _PyThreadState_NewBound(PyInterpreterState *interp, int whence)
 {
     PyThreadState *tstate = new_threadstate(interp, whence);
-    if (tstate) {
+    if (tstate && bind_claim_begin(tstate)) {
         bind_tstate(tstate);
         // This makes sure there's a gilstate tstate bound
         // as soon as possible.
         if (gilstate_get() == NULL) {
             bind_gilstate_tstate(tstate);
         }
+        bind_claim_end(tstate);
     }
+    // Otherwise finalization has taken (and cleared) the thread state before
+    // the bind: leave it unbound; the thread hangs when it tries to attach.
     return tstate;
 }
 
@@ -2190,6 +2233,9 @@
     PyThreadState *p, *next;
     for (p = list; p; p = next) {
         next = p->next;
+        if (!is_after_fork) {
+            bind_claim_for_clear(p);
+        }
         PyThreadState_Clear(p);
         tstate_delete_common(p, /*release_gil=*/0, /*is_unlinked=*/1);
         if (is_after_fork) {
@@ -2817,12 +2863,18 @@
     // was called, tstate becomes a dangling pointer.
     assert(_PyThreadState_CheckConsistency(tstate));
 
+    if (!bind_claim_begin(tstate)) {
+        // Finalization has taken the thread state before the bind: leave it
+        // unbound; the thread hangs when it tries to attach.
+        return;
+    }
     bind_tstate(tstate);
     // This makes sure there's a gilstate tstate bound
     // as soon as possible.
     if (gilstate_get() == NULL) {
         bind_gilstate_tstate(tstate);
     }
+    bind_claim_end(tstate);
 }
 
 #if defined(Py_GIL_DISABLED) && !defined(Py_LIMITED_API)

The probe in thread_run() (not part of the sketch). The pause for the C extension is the same block in _PyThreadState_NewBound(), between new_threadstate() and bind_tstate().

--- a/Modules/_threadmodule.c
+++ b/Modules/_threadmodule.c
@@ -2,6 +2,7 @@
 /* Interface to Sjoerd's portable C thread library */
 
 #include "Python.h"
+#include <unistd.h>               // usleep(), PROBE
 #include "pycore_fileutils.h"     // _PyFile_Flush
 #include "pycore_interp.h"        // _PyInterpreterState.threads.count
 #include "pycore_lock.h"
@@ -382,6 +383,12 @@ thread_run(void *boot_raw)
         goto exit;
     }
 
+    {   // PROBE: a pause at every 16th thread start
+        static int probe_count;
+        if ((_Py_atomic_add_int(&probe_count, 1) & 15) == 0) {
+            usleep(3000);
+        }
+    }
     _PyThreadState_Bind(tstate);
     PyEval_AcquireThread(tstate);
     _Py_atomic_add_ssize(&tstate->interp->threads.count, 1);

The C extension: 8 threads that loop over PyThreadState_New(), PyEval_RestoreThread(), 20 new bytes objects into a shared list (so that finalization decrefs objects they own), PyThreadState_Clear() and PyThreadState_DeleteCurrent(). It ran in the same kind of loop, with import natthreads, time; natthreads.start(8, [None] * 256); time.sleep(0.02) as the process.

#include <Python.h>
#include <pthread.h>

static PyInterpreterState *interp;
static PyObject *sink;          /* a list owned by the main thread */

static void *
worker(void *arg)
{
    for (Py_ssize_t i = (Py_ssize_t)arg;; i++) {
        PyThreadState *ts = PyThreadState_New(interp);
        PyEval_RestoreThread(ts);
        for (int k = 0; k < 20; k++) {
            PyObject *b = PyBytes_FromStringAndSize(NULL, 64);
            if (b != NULL) {
                PyList_SetItem(sink, (i * 20 + k) % PyList_GET_SIZE(sink), b);   /* steals b */
            }
        }
        PyThreadState_Clear(ts);
        PyThreadState_DeleteCurrent();
    }
    return NULL;
}

static PyObject *
start(PyObject *self, PyObject *args)
{
    int n;
    if (!PyArg_ParseTuple(args, "iO!", &n, &PyList_Type, &sink)) {
        return NULL;
    }
    Py_INCREF(sink);
    interp = PyInterpreterState_Get();
    for (int i = 0; i < n; i++) {
        pthread_t t;
        if (pthread_create(&t, NULL, worker, (void *)(Py_ssize_t)(i * 1000)) != 0) {
            return PyErr_SetFromErrno(PyExc_OSError);
        }
        pthread_detach(t);
    }
    Py_RETURN_NONE;
}

static PyMethodDef methods[] = {{"start", start, METH_VARARGS, NULL}, {NULL, NULL, 0, NULL}};
static PyModuleDef_Slot slots[] = {{Py_mod_gil, Py_MOD_GIL_NOT_USED}, {0, NULL}};
static struct PyModuleDef module = {PyModuleDef_HEAD_INIT, "natthreads", NULL, 0, methods, slots};
PyMODINIT_FUNC PyInit_natthreads(void) { return PyModuleDef_Init(&module); }

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.16.0a1+dev free-threading build (main, Nov 11 2026, 08:33:39) [GCC 14.2.0]

Activity

  1. added
    type-crashA hard crash of the interpreter, possibly with a core dump
    on Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    type-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions