I would like to propose a private API that allows CPython internals to defer automatic garbage collection on the current thread without changing the interpreter-visible GC state.
Some CPython internals build large object graphs in a burst, where a collection triggered halfway through cannot reclaim anything from that operation and only traverses a half-built graph. Using PyGC_Disable() for this changes interpreter-wide state. Saving and restoring that state is racy in the free-threaded build, and it can also be observed or changed by user code in the GIL build.
What we want is something approximately like:
_PyGC_DeferAutomaticCollection(tstate);
/* Build a large object graph. */
_PyGC_ResumeAutomaticCollection(tstate);
This would be a private CPython API. It should be per-thread and nestable, keep allocation counters running, leave gc.isenabled() unchanged, and defer only automatic collection. Leaving the outermost scope would schedule a collection if one became due.
I think there are many possible cases in the stdlib and core that can benefit from this, and we currently do not have a safe tool for them. This can be implemented with a small nesting counter in _PyThreadStateImpl, checked by the allocation paths and _Py_RunGC(). This would allow these optimizations without changing user-visible GC state or introducing cross-thread races.
Linked PRs
I would like to propose a private API that allows CPython internals to defer automatic garbage collection on the current thread without changing the interpreter-visible GC state.
Some CPython internals build large object graphs in a burst, where a collection triggered halfway through cannot reclaim anything from that operation and only traverses a half-built graph. Using
PyGC_Disable()for this changes interpreter-wide state. Saving and restoring that state is racy in the free-threaded build, and it can also be observed or changed by user code in the GIL build.What we want is something approximately like:
This would be a private CPython API. It should be per-thread and nestable, keep allocation counters running, leave
gc.isenabled()unchanged, and defer only automatic collection. Leaving the outermost scope would schedule a collection if one became due.I think there are many possible cases in the stdlib and core that can benefit from this, and we currently do not have a safe tool for them. This can be implemented with a small nesting counter in
_PyThreadStateImpl, checked by the allocation paths and_Py_RunGC(). This would allow these optimizations without changing user-visible GC state or introducing cross-thread races.Linked PRs