Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
1d93b30
Thread-safety prep for free-threading builds
greateggsgreg May 9, 2026
e029137
Initialise pythonnet on free-threaded Python (#2720)
greateggsgreg May 9, 2026
fd56aa4
Make extension/CLR-object registries thread-safe
greateggsgreg May 9, 2026
b727a15
Atomic type creation in ReflectedClrType.GetOrCreate / TypeManager.Ge…
greateggsgreg May 9, 2026
debba0d
Add free-threaded thread-stress tests and 3.14t to CI matrix
greateggsgreg May 9, 2026
585916e
Atomic GCHandle ownership and finalizer-thread shutdown guards
greateggsgreg May 10, 2026
c0ddb29
Make additional internal registries thread-safe
greateggsgreg May 10, 2026
096e466
test_thread: join worker threads before returning
greateggsgreg May 10, 2026
5005f36
test_thread: cover ModuleObject thread-safe registries
greateggsgreg May 10, 2026
3ad586a
Wider thread-safety audit fixes for free-threaded Python
greateggsgreg May 10, 2026
6b1f3f4
Document lock acquisition sites and strong->weak GCHandle swap
greateggsgreg May 10, 2026
732d97e
Preserve InternString single-write invariant under DEBUG
greateggsgreg May 10, 2026
3d56611
test_thread: cover real-world consumer patterns
greateggsgreg May 10, 2026
7094b12
Auto-detect free-threaded libpython in venv home
greateggsgreg May 12, 2026
f49ed1b
Snapshot pypath, use ConcurrentDictionary for thunks and slot holders
greateggsgreg May 13, 2026
fa89ff0
Fix handling of python runtime suffixes m/t
greateggsgreg May 13, 2026
a373e79
Fix threadtest race
greateggsgreg May 13, 2026
43a4237
Fix double-free in chained ClassDerived Finalize
greateggsgreg May 13, 2026
b42377e
Enable Mono CI jobs on free-threaded Python 3.14
greateggsgreg May 13, 2026
526a297
Inline freethreaded_only as pytest.mark.skipif at call sites
greateggsgreg May 13, 2026
8485837
Fix InterruptTest assertion on free-threaded Python 3.14
greateggsgreg May 13, 2026
a459772
Add concurrent stress tests for PyBuffer.Dispose and CLR-cycle gc.col…
greateggsgreg May 15, 2026
3bf6d68
Trim concurrent overhead on hot paths from free-threading prep
greateggsgreg May 15, 2026
3433446
Pre-warm ctor binder in concurrent-gc test to avoid first-call race
greateggsgreg May 15, 2026
4a034f7
Make MethodBinder.GetMethods lazy init thread-safe under free-threading
greateggsgreg May 15, 2026
afd5799
Precompute method precedence to avoid quadratic GetParameters allocat…
greateggsgreg May 15, 2026
95e4811
Zero the slot in ClassDerived.tp_dealloc when tp_clear already ran to…
greateggsgreg May 15, 2026
a799c44
Keep ClassDerived wrapper alive across the NewObjectToPython slot dem…
greateggsgreg May 15, 2026
220b4b4
Document private helpers added during free-threading prep
greateggsgreg May 16, 2026
dfda807
Add debug echoes and a 6-minute step timeout to the Mono test job
greateggsgreg May 16, 2026
1130f6a
Drop the per-loop CLR GC.Collect from concurrent-gc test to avoid Mon…
greateggsgreg May 16, 2026
f75cbab
Add temporary Mono-step diagnostics on Linux/macOS to locate the x64-…
greateggsgreg May 16, 2026
64ea342
Revert temporary Mono-step diagnostics now that the underlying race i…
greateggsgreg May 16, 2026
fda8211
Add user-facing threading guide covering GIL, free-threading, and com…
greateggsgreg May 18, 2026
356d76f
Harden CollectBasicObject against .NET-GC timing differences
greateggsgreg May 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
instance: macos-15
suffix: -macos-aarch64-none

python: ["3.11", "3.12", "3.13", "3.14", "3.15"]
python: ["3.11", "3.12", "3.13", "3.14", "3.14t", "3.15", "3.15t"]

exclude:
# fails to call mono methods
Expand All @@ -55,6 +55,17 @@ jobs:
platform: x86
python: '3.13'

# Free-threaded Python on Windows is not yet supported by pythonnet's
# native build chain; restrict 3.14t to Linux and macOS for now.
- os:
category: windows
platform: x86
python: ['3.14t', '3.15t']
- os:
category: windows
platform: x64
python: ['3.14t', '3.15t']

env:
NUGET_PACKAGES: ${{ github.workspace }}/.nuget/packages
steps:
Expand Down
1 change: 1 addition & 0 deletions doc/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ page. Use the `Python.NET issue tracker`_ to report issues.
python
dotnet
codecs
threading
pyreference
reference

Expand Down
214 changes: 214 additions & 0 deletions doc/source/threading.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,214 @@
Threading
=========

This page explains how Python.NET interacts with the Python Global Interpreter
Lock (GIL) and with managed threads, and what guarantees the runtime makes
when your code is multi-threaded. It covers both classic CPython builds and
the free-threaded build introduced in CPython 3.13 (``Py_GIL_DISABLED``).

The model in one paragraph
--------------------------

Python.NET embeds CPython, so every interaction with a Python object —
including reading a ``PyObject``'s attributes, calling a Python callable,
constructing a Python value, or letting a ``PyObject`` go out of scope — must
happen while the calling thread is *attached* to the interpreter. On a
classic (GIL-enabled) CPython build "attached" means "holds the GIL"; on a
free-threaded build it means "has an active thread state". In both cases the
attachment API is the same: ``Py.GIL()`` on the C# side and
``threading.Thread`` / ``_thread`` on the Python side. Forgetting to attach
will crash the process or corrupt memory.

Acquiring the GIL from C#
-------------------------

When .NET code calls into Python it must hold the GIL. Use the ``Py.GIL()``
disposable to acquire and release it::

using (Py.GIL())
{
dynamic np = Py.Import("numpy");
var arr = np.array(new[] { 1, 2, 3 });
// ... interact with arr ...
}

``Py.GIL()`` is re-entrant: nesting calls on the same thread is harmless and
cheap. Always pair acquisition with disposal — the ``using`` form does this
automatically, and you must release the GIL on the same thread that acquired
it.

If you need a Python object to outlive the ``using`` block, copy what you
need (e.g. ``.As<int[]>()`` or ``new PyObject(value)``) before releasing the
GIL.

Releasing the GIL for long-running .NET work
--------------------------------------------

If a managed call holds the GIL but then does long-running work that does not
touch Python (heavy CPU, blocking I/O, native interop), release the GIL so
other Python threads can run::

IntPtr threadState = PythonEngine.BeginAllowThreads();
try
{
DoCpuHeavyWork(); // safe: no Python C API calls
}
finally
{
PythonEngine.EndAllowThreads(threadState);
}

Inside the ``BeginAllowThreads``/``EndAllowThreads`` block you must not touch
any Python object. If you need to talk to Python from worker threads spawned
in this region, those threads must acquire the GIL themselves with
``Py.GIL()``.

Calling .NET from Python threads
--------------------------------

Calling a managed method from a Python ``threading.Thread`` works
transparently — Python.NET handles GIL acquisition/release around the
managed call. The managed code sees the GIL held on entry and is free to
release it via ``BeginAllowThreads`` if it does its own blocking work.

Calling Python from CLR threads
-------------------------------

A CLR thread that was *not* spawned by Python (a thread-pool task, a
``Thread`` started in C#, an ``async`` continuation that resumed on a
different thread, etc.) must acquire the GIL before touching any
``PyObject``::

Task.Run(() =>
{
using (Py.GIL())
{
// safe to use PyObjects here
}
});

Forgetting this is the most common pythonnet threading bug. Symptoms range
from immediate segfaults to subtle refcount corruption that crashes much
later.

Reference counting and finalizers
---------------------------------

``PyObject`` follows the .NET ``IDisposable`` pattern. ``Dispose()`` (or the
end of a ``using`` block) drops the underlying Python reference; the GC
finalizer queues the same release for the next time Python.NET is on the GIL.

Two practical consequences:

* **Don't share a single ``PyObject`` instance across threads without
serialising access.** ``PyObject`` is not internally locked. If multiple
threads concurrently dispose the same instance, the underlying refcount can
go negative.

* **Don't rely on the GC finalizer running promptly.** The PyObject is only
freed when a Python.NET API later reacquires the GIL. If your application
shuts down without that happening, finalizable PyObjects can be reported as
leaked.

Free-threaded Python (PEP 703)
------------------------------

Starting with the free-threaded CPython 3.13+ build (``Py_GIL_DISABLED``),
the GIL is no longer the serialisation point for Python C API calls.
Python.NET is tested against the ``3.14t`` (free-threaded) interpreter and
behaves as follows under that build:

* ``Py.GIL()`` still acquires a thread state. It is functionally a no-op
for mutual exclusion but is still required for thread-state attachment.
Existing code that uses ``using (Py.GIL())`` continues to work without
changes.
* ``PythonEngine.BeginAllowThreads`` / ``EndAllowThreads`` similarly
manage the thread state and are still needed if you want the GC and
other Python threads to run while you're in long-running unmanaged code.
* Internal Python.NET caches (the reflection cache, generic-type binding
cache, dynamic-dispatch cache, module attribute cache, the interned-
string table, etc.) are thread-safe. You may read and call CLR types
concurrently from any number of threads without external locking.
* The reference-counting protocol uses CPython's ``Py_REFCNT`` symbol on
3.14+, which returns the merged biased + shared refcount; values you read
from ``PyObject.Refcount`` are correct under free-threading.

Behaviour that is *unchanged* between GIL and free-threaded builds:

* A managed object exposed to Python (e.g. via ``System.Object`` or a
Python subclass of a CLR type) is still owned by a single CLR side: you
must not mutate its plain CLR fields from multiple threads without your
own locking. Python.NET only protects its own bookkeeping, not your
domain data.
* Operations on a single ``PyObject`` instance still require external
serialisation — see "Reference counting" above.

Patterns
--------

Concurrent CLR access from Python
"""""""""""""""""""""""""""""""""

Hammering CLR attributes / generic types from many threads is supported::

from threading import Thread
import System
from System.Collections.Generic import List

def worker():
for _ in range(1000):
_ = System.String.Empty
_ = List[int]()

threads = [Thread(target=worker) for _ in range(8)]
for t in threads: t.start()
for t in threads: t.join()

This works on both GIL and free-threaded builds.

Python callback invoked from a managed thread
"""""""""""""""""""""""""""""""""""""""""""""

If a managed component calls back into a Python delegate from a thread it
spawned, that callback path acquires the GIL internally — you do not need to
add ``Py.GIL()`` around the Python code in the delegate.

Spawning a managed thread from inside ``Py.GIL()``
""""""""""""""""""""""""""""""""""""""""""""""""""

If you start a managed thread while holding the GIL and the thread needs to
call back into Python, release the GIL first so the new thread can acquire
it::

using (Py.GIL())
{
var pyCallback = scope.Get("on_done");
PythonEngine.BeginAllowThreads(); // let workers acquire the GIL
try
{
// spawn workers, wait for them...
}
finally
{
PythonEngine.EndAllowThreads(...);
}
}

Without the ``BeginAllowThreads`` the spawned thread blocks forever waiting
for the GIL the parent thread is still holding.

Common pitfalls
---------------

* Holding ``Py.GIL()`` across ``Task.Run`` / ``await`` boundaries. Async
continuations can resume on a different thread; the GIL handle is
thread-bound and must be released on the same thread that acquired it.
* Passing a ``PyObject`` to a managed worker without taking ownership. If
the producer disposes its handle while the consumer is still using it,
the worker will operate on a freed object. Wrap the producer's
``PyObject`` with ``new PyObject(value)`` before handing it off, or use
``NewReference()``.
* Calling a Python callable that does CPU-bound work without releasing the
GIL. Other Python threads cannot make progress in that case, even on a
free-threaded build where the GIL is otherwise a no-op (the callable
itself may still touch contended Python state).
39 changes: 20 additions & 19 deletions src/embed_tests/TestFinalizer.cs
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ private static void FullGCCollect()
{
GC.Collect();
GC.WaitForPendingFinalizers();
GC.Collect(); // reclaim objects whose finalizers just ran
}

[Test]
Expand All @@ -51,28 +52,28 @@ public void CollectBasicObject()
Finalizer.Instance.BeforeCollect += handler;

IntPtr pyObj = MakeAGarbage(out var shortWeak, out var longWeak);
FullGCCollect();
// The object has been resurrected
Warn.If(
shortWeak.IsAlive,
"The referenced object is alive although it should have been collected",
shortWeak
);
Assert.That(
longWeak.IsAlive,
Is.True,
$"The reference object is not alive although it should still be"
);

// The real contract: after the wrapper is GC'd, the underlying
// Python pointer must end up in Finalizer's queue. Poll because
// .NET Framework / .NET Core differ in how many GC cycles it takes.
List<IntPtr> garbage = null;
for (int attempt = 0; attempt < 10; attempt++)
{
var garbage = Finalizer.Instance.GetCollectedObjects();
Assert.NotZero(garbage.Count, "There should still be garbage around");
Warn.Unless(
garbage.Contains(pyObj),
$"The {nameof(longWeak)} reference doesn't show up in the garbage list",
garbage
);
FullGCCollect();
garbage = Finalizer.Instance.GetCollectedObjects();
if (garbage.Contains(pyObj)) break;
Thread.Sleep(20);
}

Warn.If(shortWeak.IsAlive,
"shortWeak is alive after FullGCCollect; runtime hasn't reclaimed the wrapper yet",
shortWeak);
// longWeak.IsAlive at this point is .NET-GC-implementation-defined
// (Framework reclaims post-finalize objects more eagerly than Core);
// intentionally not asserted.

Assert.That(garbage, Has.Member(pyObj),
"PyObject did not reach Finalizer.Instance.GetCollectedObjects()");
try
{
Finalizer.Instance.Collect();
Expand Down
12 changes: 11 additions & 1 deletion src/embed_tests/TestInterrupt.cs
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,17 @@ import time
Assert.That(asyncCall.Wait(TimeSpan.FromSeconds(5)), Is.True, "Async thread was not interrupted in time");
PythonEngine.EndAllowThreads(threadState);

Assert.That(asyncCall.Result, Is.EqualTo(0));
// On free-threaded CPython 3.14, PyRun_SimpleString may return -1 even
// when the script catches the async-injected KeyboardInterrupt — the
// C-level error indicator depends on which bytecode boundary the
// async-exc fires at, and isn't always cleared the way GIL builds clear
// it. The interrupt firing and the script terminating cleanly are what
// this test exercises; the return code is a side-effect that's only
// deterministic under the GIL.
if (Python.Runtime.Native.ABI.IsFreeThreaded)
Assert.That(asyncCall.Result, Is.AnyOf(0, -1));
else
Assert.That(asyncCall.Result, Is.EqualTo(0));
}
}
}
5 changes: 3 additions & 2 deletions src/embed_tests/TestNativeTypeOffset.cs
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,10 @@ public class TestNativeTypeOffset
public void LoadNativeTypeOffsetClass()
{
PyObject sys = Py.Import("sys");
// We can safely ignore the "m" abi flag
// "m" is benign; "t" (free-threaded) is handled via ABI.ObjectHeadOffset
// rather than the install-time-generated NativeTypeOffset class.
var abiflags = sys.HasAttr("abiflags") ? sys.GetAttr("abiflags").ToString() : "";
abiflags = abiflags.Replace("m", "");
abiflags = abiflags.Replace("m", "").Replace("t", "");
if (!string.IsNullOrEmpty(abiflags))
{
string typeName = "Python.Runtime.NativeTypeOffset, Python.Runtime";
Expand Down
37 changes: 37 additions & 0 deletions src/embed_tests/TestPyBuffer.cs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
using System;
using System.Runtime.CompilerServices;
using System.Text;
using System.Threading;
using NUnit.Framework;
using Python.Runtime;
using Python.Runtime.Codecs;
Expand Down Expand Up @@ -129,6 +130,42 @@ public void MultidimensionalNumPyArray()
});
}

[Test]
public void ConcurrentDispose()
{
// Two threads racing on Dispose() must not double-release the view —
// Interlocked.Exchange on disposedValue gates PyBuffer_Release.
// Smoke test: no crash, exception, or buffer-protocol violation.
using var _ = Py.GIL();
using var arr = ByteArrayFromAsciiString("hello world! !$%&/()=?");

const int iterations = 200;
for (int i = 0; i < iterations; i++)
{
PyBuffer buf = arr.GetBuffer();

IntPtr ts = PythonEngine.BeginAllowThreads();
using var barrier = new Barrier(2);
Exception captured = null;
Action race = () =>
{
try
{
barrier.SignalAndWait();
using (Py.GIL()) buf.Dispose();
}
catch (Exception ex) { Interlocked.CompareExchange(ref captured, ex, null); }
};
var t1 = new Thread(() => race());
var t2 = new Thread(() => race());
t1.Start(); t2.Start();
t1.Join(); t2.Join();
PythonEngine.EndAllowThreads(ts);

if (captured != null) throw captured;
}
}

[MethodImpl(MethodImplOptions.NoInlining)]
static void MakeBufAndLeak(PyObject bufProvider)
{
Expand Down
Loading
Loading