Repository navigation
with lock: can skip __exit__ when a signal handler raises right after __enter__ #148874
Description
Bug report
A with X: statement can return without calling X.__exit__ when a Python signal handler raises between __enter__ returning and the body starting.
This leaks whatever resource __enter__ acquired (e.g. a threading.Lock stays locked forever).
Cause
In 3.14, the following changed how with X: compiles:
Before 3.14:
LOAD X
BEFORE_WITH # single op: calls enter, sets up exit on stack
POP_TOP # <- exception table starts here
... body ...
BEFORE_WITH has no periodic/eval-breaker check, so there was no window for a signal handler to fire between __enter__ returning and the with-statement's exception handler being established.
From 3.14 onward:
LOAD X
COPY 1
LOAD_SPECIAL exit
SWAP 2 / SWAP 3
LOAD_SPECIAL enter
CALL 0 # ends with _CHECK_PERIODIC_AT_END
POP_TOP # <- exception table starts here
... body ...
CALL includes _CHECK_PERIODIC_AT_END, which runs the Python signal handler. If that handler raises, JUMP_TO_LABEL(error) fires with frame->instr_ptr still pointing at CALL (set before the micro-ops run).
Reproducer (via Claude Code)
test_with_signal_leak.py
import ctypes, ctypes.util, signal, sys, threading
# signal.pthread_kill / os.kill both call PyErr_CheckSignals after the syscall,
# which perturbs timing enough to suppress the race. Use libc.pthread_kill
# directly.
_pthread_kill = ctypes.CDLL(ctypes.util.find_library("c")).pthread_kill
_pthread_kill.argtypes = [ctypes.c_ulong, ctypes.c_int]
_pthread_kill.restype = ctypes.c_int
_MAIN_TID = threading.get_ident()
def _handler(signum, frame):
raise RuntimeError("signal")
def _send():
_pthread_kill(_MAIN_TID, signal.SIGUSR1)
def run_trial(lock, iterations=200):
t = threading.Thread(target=_send)
t.start()
try:
for _ in range(iterations):
with lock:
pass
except BaseException:
pass
try:
t.join()
except BaseException:
pass
if lock.locked():
lock.release()
return True
return False
signal.signal(signal.SIGUSR1, _handler)
lock = threading.Lock()
leaked = 0
for _ in range(2000):
try:
if run_trial(lock):
leaked += 1
except BaseException:
if lock.locked():
lock.release()
print(f"leaked={leaked}/2000")cc @markshannon
Linked PRs
Reacted by Neko Asakura and Shantanu
Reacted by LincolnPuzey
Activity
markshannon commentedon Apr 23, 2026
I see a three possible fixes for this:
START_WITHinstructionCALL_NO_CHECKinstruction to replace theCALLin this caseCALLinstruction to tell it whether to do the periodic checkI don't like 1 as we would loose all interpreter specialization and JIT optimizations that we get for free by using
LOAD_SPECIALandCALL2 is OK, but adds another instruction but we can't afford to specialize it, and optimizing it in the JIT would be awkward.
3 is clearly the best for this case, but potentially slows down all other calls which outnumber the calls to
__enter__considerably. Slowing down all other calls would be bad, but I think we can avoid the slowdown:Overall, I think 3 is the best, and will allow us to skip the eval_breaker check for other calls where we don't want to allow interrupts.
colesbury commentedon Apr 23, 2026
I think it'd be good to have an option that we can backport to 3.14 without changing the bytecode:
_Py_HandlePendingto handle this case in 3.14?markshannon commentedon Apr 23, 2026
For 3.14 we could check whether the previous instruction was
LOAD_SPECIAL. There is no harm in skipping the eval_breaker on the very rare false positiveMerge branch 'main' into pythongh-148874/with-signal-exit-skip
Merge branch 'main' into pythongh-148874/with-signal-exit-skip
Merge branch 'main' into pythongh-148874/with-signal-exit-skip
Merge branch 'main' into pythongh-148874/with-signal-exit-skip
Merge branch 'main' into pythongh-148874/with-signal-exit-skip
Merge branch 'main' into pythongh-148874/with-signal-exit-skip
Merge branch 'main' into pythongh-148874/with-signal-exit-skip
Merge branch 'main' into pythongh-148874/with-signal-exit-skip
GH-148874: Make sure that mngr.__exit__() is always called in a with …
[3.15] Backport GH-148874: Make sure that mngr.__exit__() is always c…
Python release: wheel verification fixes (CPython 3.14 Ctrl+C, macOS …
Run test_interrupt in a subprocess
gamarino commentedon Nov 14, 2026
We hit this in practice on 3.14.10 and would like to ask for a 3.14 backport. GH-150911 is in main and 3.15, but not in 3.14.
In the CI of an embedded database library, pytest-timeout (signal method) raised in the main thread right after a library-internal
with lock:acquired a CRLock. The lock stayed held by the main thread, every worker thread that needed it blocked, and the job ran into the 2h30 limit. This happened on 3.14 only; the same suite passes on 3.12 and 3.13. Users are exposed through Ctrl-C (KeyboardInterrupt) in the same way.A minimal SIGALRM variant of the reproducer, with the main thread looping
with L: passand a one-shotsetitimerhandler that raises (Linux x86-64, 3000 exceptions each):RLockleft held-X gil=0-X gil=1With
threading.Lockit was 566 out of 1000 on 3.14.10, and 0 on 3.13.14.For now we document the hazard for our 3.14 users (end the process after such an interrupt) and use
timeout_method = "thread"in our tests. A 3.14 fix along the lines suggested above, without a bytecode change, would remove the need for that.sergey-miryanov commentedon Nov 14, 2026
cc @hugovk
hugovk commentedon Nov 15, 2026
Yes, this can be fixed/backported for 3.14. #150911 is already labelled "needs backport to 3.14", so is awaiting a PR.