Repository navigation
Free lists are still used after being finalized (cleared) #85064
Description
| BPO | 40887 |
|---|---|
| Nosy | @vstinner, @skrah |
| PRs |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2020-06-08.00:15:40.459>
created_at = <Date 2020-06-06.08:23:56.632>
labels = ['interpreter-core', '3.10', 'performance']
title = 'Free lists are still used after being finalized (cleared)'
updated_at = <Date 2020-06-23.09:37:03.379>
user = 'https://github.com/skrah'bugs.python.org fields:
activity = <Date 2020-06-23.09:37:03.379>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-06-08.00:15:40.459>
closer = 'vstinner'
components = ['Interpreter Core']
creation = <Date 2020-06-06.08:23:56.632>
creator = 'skrah'
dependencies = []
files = []
hgrepos = []
issue_num = 40887
keywords = ['patch']
message_count = 8.0
messages = ['370813', '370922', '370924', '370925', '370936', '370937', '370972', '372147']
nosy_count = 2.0
nosy_names = ['vstinner', 'skrah']
pr_nums = ['20698', '20700']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue40887'
versions = ['Python 3.10']
Activity
skrah commentedon Jun 6, 2020
I'm opening a separate issue to prevent bpo-40521 from getting too big.
Lists and tuples sometimes leak (starting 69ac6e5 and later):
==1445== 56 bytes in 1 blocks are definitely lost in loss record 1,542 of 4,898
==1445== at 0x4C2DE56: malloc (vg_replace_malloc.c:299)
==1445== by 0x550487: _PyObject_GC_Alloc (gcmodule.c:2233)
==1445== by 0x550487: _PyObject_GC_Malloc (gcmodule.c:2260)
==1445== by 0x550487: _PyObject_GC_New (gcmodule.c:2272)
==1445== by 0x44CB04: PyList_New (listobject.c:144)
==1445== by 0x4E3DE1: init_filters (_warnings.c:88)
==1445== by 0x4E3DE1: warnings_init_state (_warnings.c:120)
==1445== by 0x4E3DE1: _PyWarnings_InitState (_warnings.c:1372)
==1445== by 0x521720: pycore_init_import_warnings (pylifecycle.c:687)
==1445== by 0x521720: pycore_interp_init (pylifecycle.c:735)
==1445== by 0x5246A0: pyinit_config (pylifecycle.c:763)
==1445== by 0x5246A0: pyinit_core (pylifecycle.c:924)
==1445== by 0x5246A0: Py_InitializeFromConfig (pylifecycle.c:1134)
==1445== by 0x4285DC: pymain_init (main.c:66)
==1445== by 0x4296A1: pymain_main (main.c:653)
==1445== by 0x4296A1: Py_BytesMain (main.c:686)
==1445== by 0x578882F: (below main) (libc-start.c:291)
==1445== 64 bytes in 1 blocks are definitely lost in loss record 2,259 of 4,898
==1445== at 0x4C2DE56: malloc (vg_replace_malloc.c:299)
==1445== by 0x550611: _PyObject_GC_Alloc (gcmodule.c:2233)
==1445== by 0x550611: _PyObject_GC_Malloc (gcmodule.c:2260)
==1445== by 0x550611: _PyObject_GC_NewVar (gcmodule.c:2289)
==1445== by 0x48452C: tuple_alloc (tupleobject.c:76)
==1445== by 0x48452C: _PyTuple_FromArray (tupleobject.c:413)
==1445== by 0x435EE0: _PyObject_MakeTpCall (call.c:165)
==1445== by 0x436947: _PyObject_FastCallDictTstate (call.c:113)
==1445== by 0x436947: PyObject_VectorcallDict (call.c:142)
==1445== by 0x61DFC5: builtin___build_class__ (bltinmodule.c:232)
==1445== by 0x5E8A39: cfunction_vectorcall_FASTCALL_KEYWORDS (methodobject.c:440)
==1445== by 0x41F4D5: _PyObject_VectorcallTstate (abstract.h:114)
==1445== by 0x41F4D5: PyObject_Vectorcall (abstract.h:123)
==1445== by 0x41F4D5: call_function (ceval.c:5111)
==1445== by 0x42220E: _PyEval_EvalFrameDefault (ceval.c:3542)
==1445== by 0x4E6882: _PyEval_EvalFrame (pycore_ceval.h:40)
==1445== by 0x4E6882: _PyEval_EvalCode (ceval.c:4366)
==1445== by 0x4E6A65: _PyEval_EvalCodeWithName (ceval.c:4398)
vstinner commentedon Jun 7, 2020
It's not a regression. It's just that bpo-40521 helped Valgrind to detect such bug :-)
I wrote bpo-20698 to fix this issue.
I'm also working on a more generic approach to ensure that we don't add new objects to free lists after free lists are cleared.
vstinner commentedon Jun 7, 2020
I'm curious about this bug, so I looked at code changes.
Before Python 3.6, warnings.c "garbage" is declared as a static variable. It is initialized once but never cleared.
In Python 3.7, the variable moved into _PyRuntimeState.gc.
In Python 3.8, _PyGC_Fini() started to clear the "garbage" list. This function is called by Py_FinalizeEx() *after* PyList_Fini().
vstinner commentedon Jun 7, 2020
New changeset 7907f8c by Victor Stinner in branch 'master':
bpo-40887: Fix finalize_interp_clear() for free lists (GH-20698)
7907f8c
[-]Leaks in new free lists[/-][+]Free lists are still used after being finalized (cleared)[/+][-]Leaks in new free lists[/-][+]Free lists are still used after being finalized (cleared)[/+]vstinner commentedon Jun 8, 2020
New changeset bcb1983 by Victor Stinner in branch 'master':
bpo-40887: Don't use finalized free lists (GH-20700)
bcb1983
vstinner commentedon Jun 8, 2020
Thanks Stefan for the bug report. It's now fixed and I made sure that such bug cannot strike back.
skrah commentedon Jun 8, 2020
Yes, I suspected that previously reachable global objects were just unreachable after the new free lists. Thanks for the fix!
vstinner commentedon Jun 23, 2020
All free lists now have assertions to ensure that they are no longer used after their finalization.