Skip to content

bpo-38392: PyObject_GC_Track() validates object in debug mode#16615

Merged
vstinner merged 2 commits intopython:masterfrom
vstinner:gc_track_validate
Oct 7, 2019
Merged

bpo-38392: PyObject_GC_Track() validates object in debug mode#16615
vstinner merged 2 commits intopython:masterfrom
vstinner:gc_track_validate

Conversation

@vstinner
Copy link
Copy Markdown
Member

@vstinner vstinner commented Oct 7, 2019 •
edited by bedevere-bot
Loading

In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object to ensure that all object attributes visited by tp_traverse()
are initialized.

Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.

https://bugs.python.org/issue38392

@vstinner
Copy link
Copy Markdown
Member Author

vstinner commented Oct 7, 2019

This change is backward incompatible and I'm not sure that it's a good idea, so I marked is a WIP. See https://bugs.python.org/issue38392 for the rationale an discussion.

serhiy-storchaka
serhiy-storchaka approved these changes Oct 7, 2019
View reviewed changes
Copy link
Copy Markdown
Member

@serhiy-storchaka serhiy-storchaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Please add a NEWS entry.

@vstinner
Copy link
Copy Markdown
Member Author

vstinner commented Oct 7, 2019

LGTM. Please add a NEWS entry.

Should I add an entry in What's New in Python 3.9?

@serhiy-storchaka
Copy link
Copy Markdown
Member

serhiy-storchaka commented Oct 7, 2019

This change does not change the documented behavior. So I think that an entry in NEWS would be enough.

If you used PyObject_GC_Track() properly, nothing will changed for you. If you used it improperly, you should use it properly at first place.

@vstinner
bpo-38392: PyObject_GC_Track() validates object in debug mode
f587a86 
In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object type to ensure that the object is valid: test that objects
visited by tp_traverse() are valid.

Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.
@vstinner vstinner changed the title [WIP] bpo-38392: PyObject_GC_Track() validates object in debug mode bpo-38392: PyObject_GC_Track() validates object in debug mode Oct 7, 2019
@vstinner
Copy link
Copy Markdown
Member Author

vstinner commented Oct 7, 2019

I modified my PR: if a bug is detected, _PyObject_ASSERT_FAILED_MSG() now gets the object passed to PyObject_GC_Track(). This object is more likely to help the developer debugging an issue, rather than displaying the visited object which is invalid and may only be displayed as "" without any further information.

I reintroduced https://bugs.python.org/issue33803 bug with this patch:

diff --git a/Python/hamt.c b/Python/hamt.c
index 28b4e1ef6c..d7dd555d15 100644
--- a/Python/hamt.c
+++ b/Python/hamt.c
@@ -2478,8 +2478,6 @@ hamt_alloc(void)
     if (o == NULL) {
         return NULL;
     }
-    o->h_count = 0;
-    o->h_root = NULL;
     o->h_weakreflist = NULL;
     PyObject_GC_Track(o);
     return o;

The hamt bug is immediately detected. Example (with tracemalloc enabled):

$ ./python -X tracemalloc=10 -m test -v test_context
(...)
0:00:00 load avg: 0.92 [1/1] test_context
test_context_copy_1 (test.test_context.ContextTest) ...
Modules/gcmodule.c:1930: visit_validate: Assertion failed: PyObject_GC_Track() object is not valid
Memory block allocated at (most recent call first):
  File "/home/vstinner/python/master/Lib/test/test_context.py", line 322
  File "/home/vstinner/python/master/Lib/unittest/case.py", line 616
  (...)

object address  : 0x7feb39ce19b0
object refcount : 1
object type     : 0x7f5ea0
object type name: hamt
object repr     : <hamt object at 0x7feb39ce19b0>
Fatal Python error: _PyObject_AssertFailed
Python runtime state: initialized

Current thread 0x00007feb48083740 (most recent call first):
  File "/home/vstinner/python/master/Lib/test/test_context.py", line 322 in test_context_copy_1
  File "/home/vstinner/python/master/Lib/unittest/case.py", line 616 in _callTestMethod
  (...)
Aborted (core dumped)
  • The bug is detected
  • The object type and it's representation are logged
  • The Python traceback where the bug occurred is logged
  • The Python traceback where the object has been allocated is logged (this feature needs tracemalloc)

Note, when tracemalloc is disabled (default), this reminder is logged:

Enable tracemalloc to get the memory block allocation traceback

@vstinner vstinner mentioned this pull request Oct 7, 2019
Merged
@vstinner
_elementtree: fix create_elementiter()
9fa1cfe 
Onyly track the object in the GC once it's fully initialized.
@vstinner vstinner merged commit 1b18455 into python:master Oct 7, 2019
@vstinner vstinner deleted the gc_track_validate branch October 7, 2019 22:09
jacobneiltaylor pushed a commit to jacobneiltaylor/cpython that referenced this pull request Dec 5, 2019
@vstinner
bpo-38392: PyObject_GC_Track() validates object in debug mode (python…
6f8a6d6 
…GH-16615)

In debug mode, PyObject_GC_Track() now calls tp_traverse() of the
object type to ensure that the object is valid: test that objects
visited by tp_traverse() are valid.

Fix pyexpat.c: only track the parser in the GC once the parser is
fully initialized.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@serhiy-storchaka serhiy-storchaka serhiy-storchaka approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@vstinner @serhiy-storchaka @the-knights-who-say-ni @bedevere-bot