Skip to content

bpo-37210: Fix pure Python pickle: proto 5 requires _pickle.PickleBuffer#14016

Merged
vstinner merged 2 commits into
python:masterfrom
vstinner:pickle_pure_python
Jun 13, 2019
Merged

bpo-37210: Fix pure Python pickle: proto 5 requires _pickle.PickleBuffer#14016
vstinner merged 2 commits into
python:masterfrom
vstinner:pickle_pure_python

Conversation

@vstinner

@vstinner vstinner commented Jun 12, 2019 •
edited by bedevere-bot
Loading

Copy link
Copy Markdown
Member

Fix pure Python implementation of pickle: it failed with
ModuleNotFoundError on importing _pickle.PickleBuffer. If
_pickle.PickleBuffer is missing, change HIGHEST_PROTOCOL from 5 to 4
and disable PickleBuffer support.

https://bugs.python.org/issue37210

@vstinner
vstinner requested a review from pitrou June 12, 2019 14:12
@vstinner

vstinner commented Jun 12, 2019

Copy link
Copy Markdown
Member Author

@pitrou: Would you mind to review my change? Does it sound reasonable to you?

@pitrou

pitrou commented Jun 12, 2019

Copy link
Copy Markdown
Member

This sounds both fine but almost pointless to me. Is it for performance?

@vstinner

vstinner commented Jun 12, 2019

Copy link
Copy Markdown
Member Author

This sounds both fine but almost pointless to me. Is it for performance?

It repairs pyperformance, right: to measure performances of the pure Python implementation of pickle.

But other Python implementations (PyPy, MicroPython, RustPython, etc.) should also benefit of this change to reuse CPython pickle.py if they miss the _pickle module.

@pitrou

pitrou commented Jun 12, 2019

Copy link
Copy Markdown
Member

Ok. I'm not sure you need to downgrade the protocol to 4. You can just disable the code paths which depend on PickleBuffer.

@vstinner
bpo-37210: Fix pure Python implementation of pickle
701d3de 
Fix pure Python implementation of pickle: it failed with
ModuleNotFoundError on importing _pickle.PickleBuffer. If
_pickle.PickleBuffer is missing, disable PickleBuffer support.

Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
@vstinner

vstinner commented Jun 12, 2019

Copy link
Copy Markdown
Member Author

I'm not sure you need to downgrade the protocol to 4. You can just disable the code paths which depend on PickleBuffer.

Oh, I wasn't sure. I tested test_pickle with import sys; sys.modules['_pickle'] = None: I confirm that test_pickle pass if I leave HIGHEST_PROTOCOL unchanged (equal to 5).

I rebased my PR, I reverted my HIGHEST_PROTOCOL change and I made another tiny fix in test_pickle when _pickle is missing.

@vstinner

vstinner commented Jun 13, 2019

Copy link
Copy Markdown
Member Author

@pitrou: Does it look better now?

pitrou
pitrou approved these changes Jun 13, 2019
View reviewed changes

@pitrou pitrou left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1.

@vstinner
vstinner merged commit 63ab4ba into python:master Jun 13, 2019
@miss-islington

miss-islington commented Jun 13, 2019

Copy link
Copy Markdown
Contributor

Thanks @vstinner for the PR 🌮🎉.. I'm working now to backport this PR to: 3.8.
🐍🍒⛏🤖

@vstinner
vstinner deleted the pickle_pure_python branch June 13, 2019 11:58
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Jun 13, 2019
@vstinner @miss-islington
bpo-37210: Fix pure Python pickle when _pickle is unavailable (python…
f1abb42 
…GH-14016)

Allow pure Python implementation of pickle to work
even when the C _pickle module is unavailable.

Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
(cherry picked from commit 63ab4ba)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
@bedevere-bot

bedevere-bot commented Jun 13, 2019

Copy link
Copy Markdown

GH-14055 is a backport of this pull request to the 3.8 branch.

@vstinner

vstinner commented Jun 13, 2019

Copy link
Copy Markdown
Member Author

@pitrou: Thanks for the review, I updated the commit message using your NEWS entry ;-)

miss-islington added a commit that referenced this pull request Jun 13, 2019
@miss-islington @vstinner
bpo-37210: Fix pure Python pickle when _pickle is unavailable (GH-14016)
cbda40d 
Allow pure Python implementation of pickle to work
even when the C _pickle module is unavailable.

Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
(cherry picked from commit 63ab4ba)

Co-authored-by: Victor Stinner <vstinner@redhat.com>
lisroach pushed a commit to lisroach/cpython that referenced this pull request Sep 10, 2019
@vstinner @lisroach
bpo-37210: Fix pure Python pickle when _pickle is unavailable (python…
5752232 
…GH-14016)

Allow pure Python implementation of pickle to work
even when the C _pickle module is unavailable.

Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
@vstinner @DinoV
bpo-37210: Fix pure Python pickle when _pickle is unavailable (python…
5f72a32 
…GH-14016)

Allow pure Python implementation of pickle to work
even when the C _pickle module is unavailable.

Fix test_pickle when _pickle is missing: declare PyPicklerHookTests
outside "if has_c_implementation:" block.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pitrou pitrou pitrou approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@vstinner @pitrou @miss-islington @bedevere-bot @the-knights-who-say-ni