You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
multiprocessing.util._afterfork_registry leak in threaded environment #70368
When calling multiprocessing.managers.BaseManager repeatedly from a client, each time from a new thread, new entry with (already existent) multiprocessing.util.ForkAwareLocal instance is inserted as a value into multiprocessing.util._afterfork_registry dictionary on each of the calls. So the dictionary grows on each client call and may grow so indefinitely, causing memory leak if nothing else.
The attached file demonstrates the problem (Python 2.7 version, it's reproducible on 3.4 as well after 2->3 adjustments). Just run it and look at the output. The printed dictionary contains 10 entries holding the same ForkAwareLocal instance although there should be probably just one such entry there.
With master and Windows 10, the output looks like
(0, 61333912, <function ForkAwareThreadLock._reset at 0x03A783F0>): <multiprocessing.util.ForkAwareThreadLock object at 0x03A7E198>
(1, 63524512, <function ForkAwareLocal.__init__.<locals>.<lambda> at 0x03CA2940>): <multiprocessing.util.ForkAwareLocal object at 0x03C94EA0>
(2, 60787504, <function ProcessLocalSet.__init__.<locals>.<lambda> at 0x03CA2760>): ProcessLocalSet()
(5, 63524512, <function ForkAwareLocal.__init__.<locals>.<lambda> at 0x03CA2C60>): <multiprocessing.util.ForkAwareLocal object at 0x03C94EA0>
... <8 more lines like the two with ForkAwareLocal.__init__....>
I am not familiar with multiprocessing Managers and the doc is not completely clear to me. Does bug3.py only run in one process or does it result in multiple processes? If the latter, I expect it would need an "if __name__ == '__main__':" clause, at least on Windows. (Milan, I should have asked what OS you used, as mp has OS-specific behaviors.)
(https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods, Safe importing of main module)
Activity
mzamazal commentedon Jan 22, 2016
When calling multiprocessing.managers.BaseManager repeatedly from a client, each time from a new thread, new entry with (already existent) multiprocessing.util.ForkAwareLocal instance is inserted as a value into multiprocessing.util._afterfork_registry dictionary on each of the calls. So the dictionary grows on each client call and may grow so indefinitely, causing memory leak if nothing else.
The attached file demonstrates the problem (Python 2.7 version, it's reproducible on 3.4 as well after 2->3 adjustments). Just run it and look at the output. The printed dictionary contains 10 entries holding the same ForkAwareLocal instance although there should be probably just one such entry there.
terryjreedy commentedon Jan 23, 2016
Please post your 3.x version. 3.4 only gets security patches now.
mzamazal commentedon Jan 25, 2016
Python 3.x version attached.
jdemeyer commentedon Jun 13, 2019
Why is it that __init__ is called multiple times? I'm just wondering if we could somehow ensure that __init__ is called only once.
terryjreedy commentedon Jun 13, 2019
With master and Windows 10, the output looks like
(0, 61333912, <function ForkAwareThreadLock._reset at 0x03A783F0>): <multiprocessing.util.ForkAwareThreadLock object at 0x03A7E198>
(1, 63524512, <function ForkAwareLocal.__init__.<locals>.<lambda> at 0x03CA2940>): <multiprocessing.util.ForkAwareLocal object at 0x03C94EA0>
(2, 60787504, <function ProcessLocalSet.__init__.<locals>.<lambda> at 0x03CA2760>): ProcessLocalSet()
(5, 63524512, <function ForkAwareLocal.__init__.<locals>.<lambda> at 0x03CA2C60>): <multiprocessing.util.ForkAwareLocal object at 0x03C94EA0>
... <8 more lines like the two with ForkAwareLocal.__init__....>
I am not familiar with multiprocessing Managers and the doc is not completely clear to me. Does bug3.py only run in one process or does it result in multiple processes? If the latter, I expect it would need an "if __name__ == '__main__':" clause, at least on Windows. (Milan, I should have asked what OS you used, as mp has OS-specific behaviors.)
(https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods, Safe importing of main module)
mzamazal commentedon Jun 14, 2019
I used Linux.
Merge lane/sw2 b1: Window.focus() through HTML's focusing steps; Clie…