Skip to content

Importing multiprocessing breaks pickle.whichmodule #86572

Closed

Description

@renatolfc
mannequin
BPO 42406
Nosy @gpshead, @pitrou, @renatolfc, @applio, @miss-islington
PRs
  • bpo-42406: Fix whichmodule() with multiprocessing #23403
  • [3.9] bpo-42406: Fix whichmodule() with multiprocessing (GH-23403) #23560
  • [3.8] bpo-42406: Fix whichmodule() with multiprocessing (GH-23403) #23561
  • [3.7] bpo-42406: Fix whichmodule() with multiprocessing (pythonGH-23403) #28489
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/gpshead'
    closed_at = <Date 2020-11-29.23:48:33.708>
    created_at = <Date 2020-11-19.13:19:31.708>
    labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
    title = 'Importing multiprocessing breaks pickle.whichmodule'
    updated_at = <Date 2021-09-21.11:17:37.801>
    user = 'https://github.com/renatolfc'

    bugs.python.org fields:

    activity = <Date 2021-09-21.11:17:37.801>
    actor = 'python-dev'
    assignee = 'gregory.p.smith'
    closed = True
    closed_date = <Date 2020-11-29.23:48:33.708>
    closer = 'gregory.p.smith'
    components = ['Library (Lib)']
    creation = <Date 2020-11-19.13:19:31.708>
    creator = 'renatolfc'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 42406
    keywords = ['patch']
    message_count = 7.0
    messages = ['381410', '382024', '382085', '382086', '382088', '382097', '382160']
    nosy_count = 6.0
    nosy_names = ['gregory.p.smith', 'pitrou', 'renatolfc', 'python-dev', 'davin', 'miss-islington']
    pr_nums = ['23403', '23560', '23561', '28489']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'commit review'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue42406'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    Activity

    1. renatolfc commented on Nov 19, 2020

      renatolfcmannequin
      MannequinAuthor

      Importing multiprocessing prior to other modules that define ufuncs breaks pickle.whichmodule for those functions.

      Example:

          Python 3.8.6 (default, Sep 30 2020, 04:00:38)
          [GCC 10.2.0] on linux
          Type "help", "copyright", "credits" or "license" for more information.
          >>> import multiprocessing
          >>> from scipy.special import gdtrix
          >>> from pickle import whichmodule
          >>> whichmodule(gdtrix, gdtrix.__name__)
          '__mp_main__'

      Now, if we change the order of imports, whichmodule works fine:

          Python 3.8.6 (default, Sep 30 2020, 04:00:38)
          [GCC 10.2.0] on linux
          Type "help", "copyright", "credits" or "license" for more information.
          >>> from scipy.special import gdtrix
          >>> from pickle import whichmodule
          >>> import multiprocessing
          >>> whichmodule(gdtrix, gdtrix.__name__)
          'scipy.special._ufuncs'

      This happens because multiprocessing creates an alias to __main__ when imported:

      [Lib/multiprocessing/__init__.py](https://github.com/python/cpython/blob/main/Lib/multiprocessing/__init__.py)
      37:    sys.modules['__mp_main__'] = sys.modules['__main__']
      

      But doing so changes the ordering of sys.modules, and whichmodule returns __mp_main__ instead of the correct scipy.special._ufuncs.

      This bug has arisen in the context of using dill (which imports multiprocessing) for serialization and is further documented at uqfoundation/dill#392.

      I intend to submit a patch/PR with a fix soon.

    2. added
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      type-bugAn unexpected behavior, bug, or error
      stdlibStandard Library Python modules in the Lib/ directory
      and removed
      interpreter-core(Objects, Python, Grammar, and Parser dirs)
      on Nov 19, 2020
    3. gpshead commented on Nov 28, 2020

      @gpshead
      Member

      gross / nice find :)

    4. self-assigned this
      on Nov 28, 2020
    5. gpshead commented on Nov 29, 2020

      @gpshead
      Member

      New changeset 8668431 by Renato Cunha in branch 'master':
      bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
      8668431

    6. miss-islington commented on Nov 29, 2020

      @miss-islington
      Contributor

      New changeset b1c48e5 by Miss Islington (bot) in branch '3.8':
      bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
      b1c48e5

    7. miss-islington commented on Nov 29, 2020

      @miss-islington
      Contributor

      New changeset fcf7391 by Miss Islington (bot) in branch '3.9':
      bpo-42406: Fix whichmodule() with multiprocessing (GH-23403)
      fcf7391

    8. gpshead commented on Nov 29, 2020

      @gpshead
      Member

      thanks Renato!

    9. renatolfc commented on Nov 30, 2020

      renatolfcmannequin
      MannequinAuthor

      No worries, Gregory. Glad I could help. :-)

    10. transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    3.10 (EOL)end of life3.8 (EOL)end of life3.9 (EOL)end of lifestdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions