Skip to content

PyGILStateEnsure() (Probably) Crashes If Called After Finalization #124619

Open

Description

@ericsnowcurrently

Bug report

Bug description:

I'm pretty sure PyGILState_Ensure() will crash if called after finalization, particularly in the case where the thread doesn't already have a thread state active. I noticed this while looking at the function for something else.

CC @gpshead

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Activity

  1. added
    type-bugAn unexpected behavior, bug, or error
    3.12only security fixes
    3.13only security fixes
    3.14bugs and security fixes
    on Sep 26, 2024
  2. Sccucci84 commented on Sep 1, 2026

    @Sccucci84

    Hitting what looks like this same crash in real-world (non-synthetic) code, via PyGObject + GDBus, on Python 3.14.7 — so this is still unfixed on stable 3.14, not just the 3.14.0a7 build from #132948.

    Reproducer: powerprofilesctl get (a short-lived Python script shipped with power-profiles-daemon, using PyGObject to talk to power-profiles-daemon over D-Bus). It queries the current profile, prints it, and exits. Invoked frequently (once per status-bar poll interval) it segfaults intermittently with SIGSEGV / SEGV_MAPERR.

    coredumpctl info backtrace of the crashing thread (a GDBus worker thread, not the main thread):

    Stack trace of thread 284370 (gdbus):
    #0  0x00007fa616eaa612 n/a (libpython3.14.so.1.0 + 0x2aa612)
    #1  0x00007fa616e393ad PyEval_RestoreThread (libpython3.14.so.1.0 + 0x2393ad)
    #2  0x00007fa616f2a66a PyGILState_Ensure (libpython3.14.so.1.0 + 0x32a66a)
    #3  0x00007fa616a6156e n/a (_gi.cpython-314-x86_64-linux-gnu.so + 0x1156e)
    #4  0x00007fa615fd2cd9 n/a (libglib-2.0.so.0 + 0x35cd9)
    #5  0x00007fa615f5ca90 g_object_unref (libgobject-2.0.so.0 + 0x22a90)
    #6  0x00007fa614920720 n/a (libgio-2.0.so.0 + 0x10c720)
    ...
    #11 0x00007fa616001217 g_main_loop_run (libglib-2.0.so.0 + 0x64217)
    

    Frame #0 is unresolved (no debug symbols on this build), but given PyEval_RestoreThread → PyGILState_Ensure directly above it, it's almost certainly the same _PyThreadState_Attach/bind_gilstate_tstate path as the original report.

    At the same instant, the main thread is already inside exit():

    Stack trace of thread 284367 (main):
    #2  n/a (libc.so.6 + 0x40fbf)
    #3  exit (libc.so.6 + 0x4109e)
    #4  n/a (libc.so.6 + 0x27788)
    #5  __libc_start_main (libc.so.6 + 0x278b9)
    

    Mechanism appears to be: the main thread finishes and calls exit(), kicking off interpreter finalization, while a GDBus-owned worker thread is mid-g_object_unref on a proxy/connection object and its GObject finalizer calls back into Python (PyGObject's _gi), which calls PyGILState_Ensure() on a thread state that finalization has already begun tearing down.

    Reproduced twice on this machine so far (identical stack both times), a few days apart, under ordinary desktop use — no synthetic thread-spam required, just a program that talks to D-Bus from a background thread and exits promptly afterward. Given power-profiles-daemon, NetworkManager's Python bindings, and plenty of other GLib/GDBus-using Python tools follow this same "spin up async D-Bus, do one thing, exit" pattern, this is likely to affect more than just this one tool as more distros ship 3.14.

  3. russfellows commented on Oct 30, 2026

    @russfellows

    We hit this in a Rust/PyO3 extension (russfellows/s3dlio#302) and narrowed it to a pure C reproducer with no Rust or PyO3 involved. The results add something to this issue: 3.13.8+ survives this pattern and 3.14 does not, so it is a regression in 3.14, not only a long-standing API hazard.

    Reproducer

    N native (non-Python) threads loop on PyGILState_Ensure() / small Python work / PyGILState_Release(), with a short random sleep between attaches, while the main thread returns and the interpreter finalizes. Many objects with __del__ keep bytecode running during finalization.

    attacher.c (build with gcc -O2 -shared -fPIC -I$(python3 -c 'import sysconfig;print(sysconfig.get_path("include"))') attacher.c -o attacher.so):

    // Pure-CPython repro for s3dlio #302 (no s3dlio, no Rust): N native (non-Python) threads
    // repeatedly do PyGILState_Ensure / light Python work / PyGILState_Release, like tokio workers that
    // Python::attach after an await.  Every Ensure on a thread that has no thread state creates a NEW one.
    #include <Python.h>
    #include <pthread.h>
    #include <stdio.h>
    #include <stdatomic.h>
    #include <unistd.h>
    #include <stdlib.h>
    #if PY_VERSION_HEX < 0x030D0000
    #define FIN() _Py_IsFinalizing()
    #else
    #define FIN() Py_IsFinalizing()
    #endif
    static atomic_long attached_while_finalizing = 0;
    static int check_first = 0; static unsigned max_sleep = 3000;
    void set_sleep(unsigned v) { max_sleep = v; }
    void set_check(int v) { check_first = v; }
    static atomic_long attaches = 0;
    static atomic_long began_final = 0, began_final_returned = 0;
    static void *run(void *arg) {
        unsigned seed = (unsigned)(size_t)&seed ^ (unsigned)pthread_self();
        for (;;) {
            if (check_first && FIN()) { usleep(1000); continue; }   // what pyo3 Python::attach does (best-effort, racy)
            int bf = FIN();
            if (bf) atomic_fetch_add(&began_final, 1);
            PyGILState_STATE g = PyGILState_Ensure();
            if (bf) atomic_fetch_add(&began_final_returned, 1);
            if (FIN()) atomic_fetch_add(&attached_while_finalizing, 1);
            PyObject *b = PyBytes_FromStringAndSize("x", 1);   // light Python work, like Py::new(BytesView)
            Py_XDECREF(b);
            atomic_fetch_add(&attaches, 1);
            PyGILState_Release(g);
            usleep(max_sleep ? 1 + (unsigned)rand_r(&seed) % max_sleep : 0);  // I/O-like gap: threads are often NOT inside Ensure when finalization starts
        }
        return NULL;
    }
    static void report(void){ fprintf(stderr,"C-REPORT attaches=%ld attached_while_finalizing=%ld ensure_called_while_finalizing=%ld ensure_RETURNED_for_those=%ld\n", atomic_load(&attaches), atomic_load(&attached_while_finalizing), atomic_load(&began_final), atomic_load(&began_final_returned)); }
    int start(int n) {
        atexit(report);
        for (int i = 0; i < n; i++) { pthread_t t; pthread_create(&t, NULL, run, NULL); pthread_detach(t); }
        return n;
    }
    long finalizing_attaches(void) { return atomic_load(&attached_while_finalizing); }
    long total_attaches(void) { return atomic_load(&attaches); }

    drv.py (run: python3 drv.py ./attacher.so 64, where 64 is the number of native threads):

    import ctypes, sys, time, os
    lib = ctypes.CDLL(os.path.join(os.path.dirname(os.path.abspath(__file__)), sys.argv[1]))
    class D:
        def __del__(self):          # bytecode runs during finalization -> the eval breaker honours GIL-drop requests
            x = [1, 2, 3]; x.append(4)
    KEEP = [D() for _ in range(int(sys.argv[3]) if len(sys.argv) > 3 else 50000)]
    lib.set_sleep(int(os.environ.get("SLEEP","3000")))
    lib.set_check(int(os.environ.get("CHECK","0")))
    lib.start(int(sys.argv[2]) if len(sys.argv) > 2 else 8)
    time.sleep(0.5)
    print("READY", lib.total_attaches(), flush=True)

    Results (Linux x86_64 glibc, default GIL build, 64 native threads, 30 runs each)

    CPython abnormal exits
    3.14.10 23/30 and 25/30 in two batches: double free or corruption (out), SIGSEGV, Fatal Python error: PyInterpreterState_Delete: remaining threads
    3.13.18 0/30
    3.13.7, 3.12.16 0/30 (from our investigation's runs)

    Analysis (3.14.10 vs 3.13.18 sources)

    1. _Py_Finalize marks only the thread states that already exist. _PyThreadState_RemoveExcept() takes the list of thread states, and only those get _PyThreadState_SetShuttingDown() (pylifecycle.c ~2092). The comment at ~2118 notes the gap: "XXX Make sure we are preventing the creating of any new thread states".
    2. New thread states are still created. A native thread calling PyGILState_Ensure() during finalization has no thread state, so new_threadstate() creates a fresh one with no finalization check. On a default (GIL) build _PyEval_StopTheWorldAll is a no-op.
    3. In 3.14, take_gil parks only flagged thread states. It decides via _PyThreadState_MustExit(tstate), which checks the thread state's own shutting-down flag. The fresh thread state is not flagged, so the thread proceeds and later races PyInterpreterState_Delete / zapthreads. That race produces the three failure modes above:
      • a SIGSEGV in _PyThreadState_Attach while zapthreads frees the thread state;
      • a double free when the thread's own PyGILState_Release frees it too;
      • "remaining threads" when the thread state is created after zapthreads ran.
    4. 3.13.8+ asks the runtime instead. Its _PyThreadState_MustExit checks globally whether the runtime is finalizing, so the late thread state is parked too (pthread_exit & PyThread_exit_thread from PyEval_RestoreThread etc. are harmful #87135 backport), and there is no crash.

    The thread-state-based check in 3.14 appears to have lost the "late native thread" case that 3.13.8+ handles. Refusing (or parking) thread-state creation in PyGILState_Ensure once finalization has started would close it.

    Workaround on our side: no native thread attaches during shutdown; Python objects are built only on Python threads. Happy to test a patch.

  4. maikschulze commented on Nov 16, 2026

    @maikschulze
    Contributor

    I was exploring peak RSS gaps of the free-threaded build with mimalloc v3 (#159718) with overnight stress tests and ran into the same kind of crash on the free-threaded build of main, with a small C extension whose own threads keep creating thread states with PyThreadState_New() while the process exits. I came to the same cause as the comment of 2026-10-30 above. Below: how it crashes on the free-threaded build, what I saw for PyGILState_Ensure() on main, and a sketch that worked here, also with the reproducer above on 3.14. An AI assistant helped with the analysis.

    In short:

    • On 3.15 and main, the interpreter guard from PEP 788 seems to cover PyGILState_Ensure(): the reproducer above didn't fail in any of 100 runs there. PyThreadState_New() has no guard and still crashes.
    • On 3.14, a check in new_threadstate() (sketch below) made the reproducer pass: 0 of 100 runs, against 38 (default build) and 79 (free-threaded) without it.

    On the free-threaded build

    The extension's 8 threads loop over PyThreadState_New(), PyEval_RestoreThread(), a little Python work, PyThreadState_Clear() and PyThreadState_DeleteCurrent() (code in the fold of #162109). The process exits after 20 ms. In a loop of fresh processes on main at e8c0584 (2026-11-10), Linux arm64, release build, 337 of 12,288 exits crashed (SIGSEGV), in finalization's interpreter_clear() while it clears another thread's thread state (backtrace trimmed):

    #0  llist_remove (node=0x264ea014b68) at Include/internal/pycore_llist.h:83
    #1  _Py_brc_remove_thread (tstate=0x264ea010010) at Python/brc.c:220
    #2  PyThreadState_Clear (tstate=0x264ea010010) at Python/pystate.c:1967
    #3  interpreter_clear (interp=0xaaaabe12fa30 <_PyRuntime+160968>, tstate=0xaaaabe150708 <_PyRuntime+295328>) at Python/pystate.c:918
    #6  _Py_Finalize (runtime=0xaaaabe108568 <_PyRuntime>) at Python/pylifecycle.c:2584
    

    In the three core files I looked at, the interpreter's list of thread states holds, besides the finalizing thread's, only the newest one or two, so they were created after finalization had taken the others out. The one being cleared belongs to a thread that is attached and running; the older threads hang in _PyThreadState_HangThread(), as they should.

    The change in _PyThreadState_MustExit() that the comment of 2026-10-30 describes came with #130649, the fix for #124878. If I read the documentation right, such a thread should hang: calling PyEval_RestoreThread() while the runtime is finalizing "will hang the thread until the program exits, even if the thread was not created by Python".

    PyGILState_Ensure() on main

    On 3.15 and main, PyGILState_Ensure() holds an interpreter guard from PEP 788 (#149116) while it creates and binds its thread state, and finalization waits until no guard is held before it takes the thread states out. As far as I can see, that covers it there: on main, release builds, the reproducer above didn't fail in any of 100 runs, on the default and on the free-threaded build, and the same extension with PyGILState_Ensure()/PyGILState_Release() instead of PyThreadState_New() didn't crash in 12,868 exits. 3.14, which the reports above are about, doesn't have the guard. PyThreadState_New() has no guard on main either.

    A sketch that worked here

    I'm not sure this is the right place for a fix; it is what worked in my tests. In new_threadstate(): when the interpreter is finalizing and the caller isn't the finalizing thread, the new thread state is treated like the ones finalization took out. It is marked as shutting down and not added to the interpreter's list, so its thread hangs when it tries to attach. Two things turned out to matter:

    • Keeping it out of the list. Marking it but leaving it in the list (what I tried first) made the exit hang: a stop-the-world (the final garbage collection has one) waits for every thread state in the list to pause, and one that is shutting down never does. That hung 5 of 2,651 exits of a release build, and a debug build aborts on the assertion in _PyThreadState_ResumeDetached().
    • Checking before the TLBC index and the QSBR slot are reserved. PyInterpreterState_Delete() frees both at the very end of finalization, while such threads may still be creating thread states. With the check placed after the reservations, a release build still crashed in 82 of 47,214 exits; the 4 core files I kept all crash in heap_pop() of the TLBC index pool. A thread state created this late never attaches, so it needs neither.

    The sketch also makes tstate_delete_common() unlink a thread state only if it is in the list. A thread state created this late is leaked: its thread hangs, and nothing else frees it.

    main, free-threaded release build exits crashed
    as it is 12,288 337
    with the sketch 12,783 0

    The full test suite passes with it on Linux (free-threaded release build), and together with the sketch in #162109 also on free-threaded debug and default builds. On main's default build the extension didn't crash (0 of 4,446 exits as it is, 0 of 8,712 with both sketches). A debug build still aborts now and then, for two other reasons: the window in that issue (a thread state cleared while its thread is still binding it), and an assertion in mimalloc that shows up without finalization too.

    On 3.14

    Here PyGILState_Ensure() also creates its thread state with new_threadstate() and has no guard, so the same check covers it. I ported the sketch to the 3.14 branch at 8aa5d65 (2026-11-10); it differs only where 3.14's code does, in the order of the two reservations and in the stop-the-world check in tstate_delete_common(). Release builds, Linux:

    on 3.14 runs as it is failed runs with the sketch failed
    the reproducer above, default build 100 38 100 0
    the reproducer above, free-threaded build 100 79 100 0
    the extension with PyThreadState_New(), default build 8,750 2 8,794 0
    the extension with PyThreadState_New(), free-threaded build 8,579 259 8,584 0
    the extension with PyGILState_Ensure(), default build 8,728 6 8,820 0
    the extension with PyGILState_Ensure(), free-threaded build 8,544 240 8,500 1

    The one failure with the sketch is an abort whose output mentions a freed thread state. I haven't looked into it; it may be the window of #162109, since PyGILState_Ensure() on 3.14 binds its new thread state itself, outside both sketches.

    With the 3.14 port, the full test suite passes on 3.14 too (default and free-threaded builds). Three test files that compile a script into a zip file failed in my runs with and without the sketch, because I had PYTHONPYCACHEPREFIX set; without it they pass.

    I may well be missing context here, and I'm glad to test other approaches.

    The sketch (against e8c0584, 2026-11-10)
    --- a/Python/pystate.c
    +++ b/Python/pystate.c
    @@ -1738,6 +1738,16 @@
         interp->threads.head = tstate;
     }
     
    +// Finalization has taken the other thread states out of the interpreter
    +// (_PyThreadState_RemoveExcept()) and counts on no new ones being added.
    +static int
    +finalization_took_threads(PyInterpreterState *interp)
    +{
    +    return (_PyInterpreterState_GetFinalizing(interp) != NULL &&
    +            _PyInterpreterState_GetFinalizingID(interp)
    +                != PyThread_get_thread_ident());
    +}
    +
     static PyThreadState *
     new_threadstate(PyInterpreterState *interp, int whence)
     {
    @@ -1755,17 +1765,24 @@
             return NULL;
         }
     #endif
    +    // A thread state created this late never attaches: it needs no TLBC
    +    // index nor QSBR slot (which the end of finalization frees).
    +    int late = finalization_took_threads(interp);
     #ifdef Py_GIL_DISABLED
    -    int32_t tlbc_idx = _Py_ReserveTLBCIndex(interp);
    -    if (tlbc_idx < 0) {
    -        free_threadstate(tstate);
    -        return NULL;
    -    }
    -    Py_ssize_t qsbr_idx = _Py_qsbr_reserve(interp);
    -    if (qsbr_idx < 0) {
    -        _Py_UnreserveTLBCIndex(interp, tlbc_idx);
    -        free_threadstate(tstate);
    -        return NULL;
    +    int32_t tlbc_idx = -1;
    +    Py_ssize_t qsbr_idx = -1;
    +    if (!late) {
    +        tlbc_idx = _Py_ReserveTLBCIndex(interp);
    +        if (tlbc_idx < 0) {
    +            free_threadstate(tstate);
    +            return NULL;
    +        }
    +        qsbr_idx = _Py_qsbr_reserve(interp);
    +        if (qsbr_idx < 0) {
    +            _Py_UnreserveTLBCIndex(interp, tlbc_idx);
    +            free_threadstate(tstate);
    +            return NULL;
    +        }
         }
     #endif
     
    @@ -1777,16 +1794,29 @@
         uint64_t id = interp->threads.next_unique_id;
         init_threadstate(tstate, interp, id, whence);
     
    -    // Add the new thread state to the interpreter.
    -    PyThreadState *old_head = interp->threads.head;
    -    add_threadstate(interp, (PyThreadState *)tstate, old_head);
    +    // Checked again under the lock that _PyThreadState_RemoveExcept() takes.
    +    late = late || finalization_took_threads(interp);
    +    if (late) {
    +        // Treat it like the thread states that finalization took out: keep
    +        // it out of the interpreter's list and mark it as shutting down, so
    +        // that its thread hangs when it tries to attach.
    +        ((PyThreadState *)tstate)->state = _Py_THREAD_SHUTTING_DOWN;
    +    }
    +    else {
    +        // Add the new thread state to the interpreter.
    +        PyThreadState *old_head = interp->threads.head;
    +        add_threadstate(interp, (PyThreadState *)tstate, old_head);
    +    }
     
         HEAD_UNLOCK(interp->runtime);
     
     #ifdef Py_GIL_DISABLED
    -    // Must be called with lock unlocked to avoid lock ordering deadlocks.
    -    _Py_qsbr_register(tstate, interp, qsbr_idx);
    -    tstate->tlbc_index = tlbc_idx;
    +    if (!late) {
    +        // Must be called with lock unlocked to avoid lock ordering deadlocks.
    +        _Py_qsbr_register(tstate, interp, qsbr_idx);
    +        tstate->tlbc_index = tlbc_idx;
    +    }
    +    // A reservation made before finalization started is left unused.
     #endif
     
         return (PyThreadState *)tstate;
    @@ -2023,17 +2053,20 @@
         _PyRuntimeState *runtime = interp->runtime;
     
         HEAD_LOCK(runtime);
    +    // A thread state created during finalization isn't in the list.
    +    int linked = (tstate->prev != NULL || interp->threads.head == tstate);
         if (tstate->prev) {
             tstate->prev->next = tstate->next;
         }
    -    else {
    +    else if (linked) {
             interp->threads.head = tstate->next;
         }
         if (tstate->next) {
             tstate->next->prev = tstate->prev;
         }
         int state = _Py_atomic_load_int_relaxed(&tstate->state);
    -    if (state != _Py_THREAD_SUSPENDED &&
    +    if (linked &&
    +        state != _Py_THREAD_SUSPENDED &&
             state != _Py_THREAD_SUSPENDED_WAITING)
         {
             // Any ongoing stop-the-world request should not wait for us because
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixes3.13only security fixes3.14bugs and security fixestopic-C-APItype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions