Skip to content

multiprocessing.util._afterfork_registry leak in threaded environment #70368

Open

Description

@mzamazal
mannequin
BPO 26180
Nosy @terryjreedy, @nirs, @jdemeyer, @applio
PRs
  • gh-70368: Fix multiple registration of ForkAwareLocal atfork cleaner #13986
  • Files
  • bug.py
  • bug3.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2016-01-22.15:24:46.852>
    labels = ['library', '3.9', 'performance']
    title = 'multiprocessing.util._afterfork_registry leak in threaded environment'
    updated_at = <Date 2019-06-14.07:05:58.650>
    user = 'https://bugs.python.org/mzamazal'

    bugs.python.org fields:

    activity = <Date 2019-06-14.07:05:58.650>
    actor = 'mzamazal'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2016-01-22.15:24:46.852>
    creator = 'mzamazal'
    dependencies = []
    files = ['41690', '41706']
    hgrepos = []
    issue_num = 26180
    keywords = ['patch']
    message_count = 6.0
    messages = ['258816', '258850', '258896', '345473', '345524', '345561']
    nosy_count = 5.0
    nosy_names = ['terry.reedy', 'nirs', 'jdemeyer', 'davin', 'mzamazal']
    pr_nums = ['13986']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'resource usage'
    url = 'https://bugs.python.org/issue26180'
    versions = ['Python 3.9']

    Activity

    1. mzamazal commented on Jan 22, 2016

      mzamazalmannequin
      MannequinAuthor

      When calling multiprocessing.managers.BaseManager repeatedly from a client, each time from a new thread, new entry with (already existent) multiprocessing.util.ForkAwareLocal instance is inserted as a value into multiprocessing.util._afterfork_registry dictionary on each of the calls. So the dictionary grows on each client call and may grow so indefinitely, causing memory leak if nothing else.

      The attached file demonstrates the problem (Python 2.7 version, it's reproducible on 3.4 as well after 2->3 adjustments). Just run it and look at the output. The printed dictionary contains 10 entries holding the same ForkAwareLocal instance although there should be probably just one such entry there.

    2. added
      stdlibStandard Library Python modules in the Lib/ directory
      performancePerformance or resource usage
      on Jan 22, 2016
    3. terryjreedy commented on Jan 23, 2016

      @terryjreedy
      Member

      Please post your 3.x version. 3.4 only gets security patches now.

    4. mzamazal commented on Jan 25, 2016

      mzamazalmannequin
      MannequinAuthor

      Python 3.x version attached.

    5. jdemeyer commented on Jun 13, 2019

      @jdemeyer
      Contributor

      Why is it that __init__ is called multiple times? I'm just wondering if we could somehow ensure that __init__ is called only once.

    6. terryjreedy commented on Jun 13, 2019

      @terryjreedy
      Member

      With master and Windows 10, the output looks like
      (0, 61333912, <function ForkAwareThreadLock._reset at 0x03A783F0>): <multiprocessing.util.ForkAwareThreadLock object at 0x03A7E198>
      (1, 63524512, <function ForkAwareLocal.__init__.<locals>.<lambda> at 0x03CA2940>): <multiprocessing.util.ForkAwareLocal object at 0x03C94EA0>
      (2, 60787504, <function ProcessLocalSet.__init__.<locals>.<lambda> at 0x03CA2760>): ProcessLocalSet()
      (5, 63524512, <function ForkAwareLocal.__init__.<locals>.<lambda> at 0x03CA2C60>): <multiprocessing.util.ForkAwareLocal object at 0x03C94EA0>
      ... <8 more lines like the two with ForkAwareLocal.__init__....>

      I am not familiar with multiprocessing Managers and the doc is not completely clear to me. Does bug3.py only run in one process or does it result in multiple processes? If the latter, I expect it would need an "if __name__ == '__main__':" clause, at least on Windows. (Milan, I should have asked what OS you used, as mp has OS-specific behaviors.)
      (https://docs.python.org/3/library/multiprocessing.html#the-spawn-and-forkserver-start-methods, Safe importing of main module)

    7. mzamazal commented on Jun 14, 2019

      mzamazalmannequin
      MannequinAuthor

      I used Linux.

    8. transferred this issue fromon Apr 10, 2022
    9. added
      type-bugAn unexpected behavior, bug, or error
      and removed
      performancePerformance or resource usage
      on Aug 31, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or error

      Projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions