_pickle.PicklingError on 3.14.0a1+ where it didn't before due to the start method change from fork to forkserver

Bug report

Bug description:

Code dating back to Python 2 had to do super(OwnClassName, self) instead of super(). Modules designed to be reloadable could not use the super(OwnClassName, self) syntax in their non-constructor method, as OwnClassName was re-binded to the class of the new module, rather than the class that had the name when the object was constructed (which differs when the object was constructed before a reload). Therefore, such modules defined classes like this:

class Subclass(Superclass):
    def __init__(self):
        self.__parent = super(Subclass, self)

    def method1(self, arg):
        self.__parent.method1(arg + 2)

Python 3.14.0a1+ breaks existing code using this trick on subclasses of multiprocessing.Process.

For example:

import multiprocessing

class MyProcess(multiprocessing.Process):
    def __init__(self, target=None, args=(), kwargs={}):
        self.__parent = super(MyProcess, self)
        self.__parent.__init__(target=target, args=args, kwargs=kwargs)

    def run(self):
        self.__parent.run()
    
if __name__ == "__main__":
    p = MyProcess()
    p.start()

worked fine on previous versions, but now errors with:

Traceback (most recent call last):
  File "/home/dev-irc/Limnoria/repro.py", line 12, in <module>
    p.start()
    ~~~~~~~^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ~~~~~~~~~~~^^^^^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/context.py", line 300, in _Popen
    return Popen(process_obj)
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/popen_forkserver.py", line 35, in __init__
    super().__init__(process_obj)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
    ~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/popen_forkserver.py", line 47, in _launch
    reduction.dump(process_obj, buf)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home/dev-irc/.local-py-git/lib/python3.14/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
_pickle.PicklingError: first argument to __newobj__() must be <class 'super'>, not <class '__main__.MyProcess'>
when serializing super object
when serializing dict item '_MyProcess__parent'
when serializing MyProcess state
when serializing MyProcess object

Full version string: Python 3.14.0a1+ (heads/main:c8fd4b12e3d, Oct 18 2024, 22:51:39) [GCC 12.2.0] on linux

CPython versions tested on:

3.14, CPython main branch

Operating systems tested on:

Linux

Linked PRs

gh-125714: Doc: Explain that multiprocessing only supports unpicklable functions through the fork method #125750

Your repro case also fails for me on 3.11 through 3.13. Simpler code that fails similarly:

import pickle
class X: pass
s = super(X, X())
pickle.dumps(s)

Strange, my repro case ~~fails~~ passes on 3.11.2; and it's based on a codebase that passed on 3.12 and 3.13 so far.

Yours does fail on 3.11.2 too, though.

Can you try this?

import multiprocessing

def f():
    def g():
        print("hello")

    multiprocessing.Process(target=g).start()
    
if __name__ == "__main__":
    f()

For me it passes on 3.11.2, but fails on 3.14.0a1+, even though function g is unpicklable no matter the version.

@gpshead

Oh, this is because I am on MacOS. The relevant change is listed in https://docs.python.org/3.14/whatsnew/3.14.html#deprecated: the default start method moved from fork to forkserver on Linux. With fork, I guess we didn't need things to be picklable.

cc @gpshead as the author of this change, but I think this will be a won't fix: We made that change after careful consideration, and this sort of issue was a known consequence. If you want the old behavior, you can either explicitly use the fork start method or refactor your code so the relevant objects can be pickled.

There might be room for improvements in how super objects are pickled, though.

Ah, I understand. Thanks!

I feel there is some room for improvement regarding documentation here, as this is a breaking change and not a deprecation as far as I can tell. And neither the changelog nor "Contexts and start methods" mentions only fork supports unpicklable Process attributes or targets

@progval

Agreed, this is mostly a documentation issue. But I don't have a good feel for exactly what to state. Care to draft up some text in a PR @progval? I think we'd want this to be clarified in both what's new 3.14 and multiprocessing docs.

Done in #125750

@serhiy-storchaka

CC: @serhiy-storchaka

@progval

@progval If you're interested, feel free to open an issue specifically about pickling super objects. I couldn't find any existing issue about that, and the code for super doesn't have anything geared towards pickling support. It's possible that we could fix this to make super objects properly picklable.

>>> import pickle
>>> class X: pass
... 
>>> pickle.dumps(super(X, X()))
Traceback (most recent call last):
  File "<python-input-6>", line 1, in <module>
    pickle.dumps(super(X, X()))
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^
_pickle.PicklingError: first argument to __newobj__() must be <class 'super'>, not <class '__main__.X'>
when serializing super object

Created a separate issue for pickling super: #125767.

Given the user reports (now plural) on this one, we should also catch and improve the error message within multiprocessing when pickle fails to at least point people in the right direction.

Avoid the multiprocessing forkserver method

Avoid the multiprocessing forkserver method (#656)

tests: fix "can't pickle local object" errors on Python 3.14

devel: handle multiprocessing regression in Python 3.14

Navigation Menu

Repository navigation

`_pickle.PicklingError` on 3.14.0a1+ where it didn't before due to the start method change from fork to forkserver #125714

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

_pickle.PicklingError on 3.14.0a1+ where it didn't before due to the start method change from fork to forkserver #125714

Description

Bug report

Bug description:

CPython versions tested on:

Operating systems tested on:

Linked PRs

Activity

JelleZijlstra commented on Oct 18, 2024

progval commented on Oct 18, 2024

progval commented on Oct 18, 2024

JelleZijlstra commented on Oct 18, 2024

progval commented on Oct 19, 2024

gpshead commented on Oct 19, 2024

progval commented on Oct 20, 2024

furkanonder commented on Oct 20, 2024

JelleZijlstra commented on Oct 21, 2024

serhiy-storchaka commented on Oct 21, 2024

gpshead commented on Nov 14, 2024

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

`_pickle.PicklingError` on 3.14.0a1+ where it didn't before due to the start method change from fork to forkserver #125714