Skip to content

Commit 28f41a5

Browse files
vstinnervstinner
committed
gh-108634: Py_TRACE_REFS uses a hash table
Python built with "configure --with-trace-refs" (tracing references) is now ABI compatible with Python release build and debug build. Moreover, it now also supports the Limited API. Change Py_TRACE_REFS build: * Remove _PyObject_EXTRA_INIT macro. * The PyObject structure no longer has two extra members (_ob_prev and _ob_next). * Use a hash table (_Py_hashtable_t) to trace references (all objects): PyInterpreterState.object_state.refchain. * Py_TRACE_REFS build is now ABI compatible with release build and debug build. * Limited C API extensions can now be built with Py_TRACE_REFS: xxlimited, xxlimited_35, _testclinic_limited. * No longer rename PyModule_Create2() and PyModule_FromDefAndSpec2() functions to PyModule_Create2TraceRefs() and PyModule_FromDefAndSpec2TraceRefs(). * _Py_PrintReferenceAddresses() is now called before finalize_interp_delete() which deletes the refchain hash table. Test changes for Py_TRACE_REFS: * Add test.support.Py_TRACE_REFS constant. * Add test_sys.test_getobjects() to test sys.getobjects() function. * test_exceptions skips test_recursion_normalizing_with_no_memory() and test_memory_error_in_PyErr_PrintEx() if Python is built with Py_TRACE_REFS. * test_repl skips test_no_memory(). * test_capi skisp test_set_nomemory().
1 parent c7cef54 commit 28f41a5

31 files changed

+269
-245
lines changed

‎Doc/c-api/typeobj.rst‎

Lines changed: 0 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -528,28 +528,6 @@ type objects) *must* have the :c:member:`~PyVarObject.ob_size` field.
528528
This field is inherited by subtypes.
529529

530530

531-
.. c:member:: PyObject* PyObject._ob_next
532-
PyObject* PyObject._ob_prev
533-
534-
These fields are only present when the macro ``Py_TRACE_REFS`` is defined
535-
(see the :option:`configure --with-trace-refs option <--with-trace-refs>`).
536-
537-
Their initialization to ``NULL`` is taken care of by the
538-
``PyObject_HEAD_INIT`` macro. For :ref:`statically allocated objects
539-
<static-types>`, these fields always remain ``NULL``. For :ref:`dynamically
540-
allocated objects <heap-types>`, these two fields are used to link the
541-
object into a doubly linked list of *all* live objects on the heap.
542-
543-
This could be used for various debugging purposes; currently the only uses
544-
are the :func:`sys.getobjects` function and to print the objects that are
545-
still alive at the end of a run when the environment variable
546-
:envvar:`PYTHONDUMPREFS` is set.
547-
548-
**Inheritance:**
549-
550-
These fields are not inherited by subtypes.
551-
552-
553531
PyVarObject Slots
554532
-----------------
555533

‎Doc/using/configure.rst‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -408,8 +408,7 @@ See also the :ref:`Python Development Mode <devmode>` and the
408408
.. versionchanged:: 3.8
409409
Release builds and debug builds are now ABI compatible: defining the
410410
``Py_DEBUG`` macro no longer implies the ``Py_TRACE_REFS`` macro (see the
411-
:option:`--with-trace-refs` option), which introduces the only ABI
412-
incompatibility.
411+
:option:`--with-trace-refs` option).
413412

414413

415414
Debug options
@@ -430,8 +429,14 @@ Debug options
430429
* Add :func:`sys.getobjects` function.
431430
* Add :envvar:`PYTHONDUMPREFS` environment variable.
432431

433-
This build is not ABI compatible with release build (default build) or debug
434-
build (``Py_DEBUG`` and ``Py_REF_DEBUG`` macros).
432+
The :envvar:`PYTHONDUMPREFS` environment variable can be used to dump
433+
objects and reference counts still alive at Python exit.
434+
435+
:ref:`Statically allocated objects <static-types>` are not traced.
436+
437+
.. versionchanged:: 3.13
438+
This build is now ABI compatible with release build and :ref:`debug build
439+
<debug-build>`.
435440

436441
.. versionadded:: 3.8
437442

‎Doc/whatsnew/3.13.rst‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -828,6 +828,11 @@ Build Changes
828828
* SQLite 3.15.2 or newer is required to build the :mod:`sqlite3` extension module.
829829
(Contributed by Erlend Aasland in :gh:`105875`.)
830830

831+
* Python built with :file:`configure` :option:`--with-trace-refs` (tracing
832+
references) is now ABI compatible with Python release build and
833+
:ref:`debug build <debug-build>`.
834+
(Contributed by Victor Stinner in :gh:`108634`.)
835+
831836

832837
C API Changes
833838
=============
@@ -900,6 +905,10 @@ New Features
900905
(with an underscore prefix).
901906
(Contributed by Victor Stinner in :gh:`108014`.)
902907

908+
* Python built with :file:`configure` :option:`--with-trace-refs` (tracing
909+
references) now supports the :ref:`Limited API <limited-c-api>`.
910+
(Contributed by Victor Stinner in :gh:`108634`.)
911+
903912
Porting to Python 3.13
904913
----------------------
905914

‎Include/internal/pycore_global_objects_fini_generated.h‎

Lines changed: 0 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_global_strings.h‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -276,10 +276,8 @@ struct _Py_global_strings {
276276
STRUCT_FOR_ID(after_in_child)
277277
STRUCT_FOR_ID(after_in_parent)
278278
STRUCT_FOR_ID(aggregate_class)
279-
STRUCT_FOR_ID(alias)
280279
STRUCT_FOR_ID(append)
281280
STRUCT_FOR_ID(argdefs)
282-
STRUCT_FOR_ID(args)
283281
STRUCT_FOR_ID(arguments)
284282
STRUCT_FOR_ID(argv)
285283
STRUCT_FOR_ID(as_integer_ratio)
@@ -403,8 +401,6 @@ struct _Py_global_strings {
403401
STRUCT_FOR_ID(errors)
404402
STRUCT_FOR_ID(event)
405403
STRUCT_FOR_ID(eventmask)
406-
STRUCT_FOR_ID(exc_type)
407-
STRUCT_FOR_ID(exc_value)
408404
STRUCT_FOR_ID(excepthook)
409405
STRUCT_FOR_ID(exception)
410406
STRUCT_FOR_ID(existing_file_name)
@@ -720,7 +716,6 @@ struct _Py_global_strings {
720716
STRUCT_FOR_ID(timetuple)
721717
STRUCT_FOR_ID(top)
722718
STRUCT_FOR_ID(trace_callback)
723-
STRUCT_FOR_ID(traceback)
724719
STRUCT_FOR_ID(trailers)
725720
STRUCT_FOR_ID(translate)
726721
STRUCT_FOR_ID(true)

‎Include/internal/pycore_object.h‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,6 @@ PyAPI_FUNC(int) _PyObject_IsFreed(PyObject *);
4949
backwards compatible solution */
5050
#define _PyObject_HEAD_INIT(type) \
5151
{ \
52-
_PyObject_EXTRA_INIT \
5352
.ob_refcnt = _Py_IMMORTAL_REFCNT, \
5453
.ob_type = (type) \
5554
},
@@ -178,6 +177,8 @@ _PyType_HasFeature(PyTypeObject *type, unsigned long feature) {
178177
extern void _PyType_InitCache(PyInterpreterState *interp);
179178

180179
extern void _PyObject_InitState(PyInterpreterState *interp);
180+
extern void _PyObject_FiniState(PyInterpreterState *interp);
181+
extern bool _PyRefchain_IsTraced(PyInterpreterState *interp, PyObject *obj);
181182

182183
/* Inline functions trading binary compatibility for speed:
183184
_PyObject_Init() is the fast version of PyObject_Init(), and

‎Include/internal/pycore_object_state.h‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ extern "C" {
88
# error "this header requires Py_BUILD_CORE define"
99
#endif
1010

11+
#include "pycore_hashtable.h" // _Py_hashtable_t
12+
1113
struct _py_object_runtime_state {
1214
#ifdef Py_REF_DEBUG
1315
Py_ssize_t interpreter_leaks;
@@ -20,11 +22,10 @@ struct _py_object_state {
2022
Py_ssize_t reftotal;
2123
#endif
2224
#ifdef Py_TRACE_REFS
23-
/* Head of circular doubly-linked list of all objects. These are linked
24-
* together via the _ob_prev and _ob_next members of a PyObject, which
25-
* exist only in a Py_TRACE_REFS build.
26-
*/
27-
PyObject refchain;
25+
// Hash table storing all objects. The key is the object pointer
26+
// (PyObject*) and the value is always the number 1 (as uintptr_t).
27+
// See _PyRefchain_IsTraced() and _PyRefchain_Trace() functions.
28+
_Py_hashtable_t *refchain;
2829
#endif
2930
int _not_used;
3031
};

‎Include/internal/pycore_runtime_init.h‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -192,7 +192,7 @@ extern PyTypeObject _PyExc_MemoryError;
192192
#ifdef Py_TRACE_REFS
193193
# define _py_object_state_INIT(INTERP) \
194194
{ \
195-
.refchain = {&INTERP.object_state.refchain, &INTERP.object_state.refchain}, \
195+
.refchain = NULL, \
196196
}
197197
#else
198198
# define _py_object_state_INIT(INTERP) \

‎Include/internal/pycore_runtime_init_generated.h‎

Lines changed: 0 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Include/internal/pycore_unicodeobject_generated.h‎

Lines changed: 0 additions & 15 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)