Crash when a callable iterator is shared between threads #161792

Crash report

What happened?

In the free-threaded build, an iter(callable, sentinel) iterator that is used from several threads can crash the interpreter.

calliter_iternext() calls it_callable and compares the result against it_sentinel through borrowed pointers. Another thread can concurrently clear those fields when the iterator is exhausted, or replace it_sentinel / it_stop_exc in __setstate__(), freeing an object that is still in use. Two threads that exhaust the iterator at the same time can also both Py_DECREF the same callable and sentinel.

Reproducer (segfaults within a few seconds on a free-threaded release build of main):

import threading

N = 2000

def worker(its, barrier):
    barrier.wait()
    for it in its:
        for _ in it:
            pass

for _ in range(200):
    # Each iterator holds the only reference to its callable.
    its = [iter(lambda: None, None) for _ in range(N)]
    barrier = threading.Barrier(4)
    threads = [threading.Thread(target=worker, args=(its, barrier))
               for _ in range(4)]
    for t in threads:
        t.start()
    for t in threads:
        t.join()

The default (GIL) build is not affected.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

• gh-161792: Fix crashes in the callable iterator #161793