Skip to content

bpo-35715: Liberate return value of _process_worker#11514

Merged
pablogsal merged 3 commits into
python:masterfrom
dchevell:master
Mar 16, 2019
Merged

bpo-35715: Liberate return value of _process_worker#11514
pablogsal merged 3 commits into
python:masterfrom
dchevell:master

Conversation

@dchevell

@dchevell dchevell commented Jan 11, 2019 •
edited
Loading

Copy link
Copy Markdown
Contributor

ProcessPoolExecutor workers will hold the return value of their last task in memory until the next task is received. Since the return value has already been propagated to the parent process's Future or else effectively discarded by this point, the memory can be safely released.

Simple case to reproduce:

import concurrent.futures
import time

executor = concurrent.futures.ProcessPoolExecutor(max_workers=1)

def big_val():
    return [{1:1} for i in range(1, 1000000)]

executor.submit(big_val)

# Observe the memory usage of the process worker during the sleep interval
time.sleep(10)

This should be easily addressed by having the worker explicitly del r after calling _sendback_result as it already does this for call_item.

https://bugs.python.org/issue35715

@the-knights-who-say-ni

the-knights-who-say-ni commented Jan 11, 2019

Copy link
Copy Markdown

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept your contribution by verifying you have signed the PSF contributor agreement (CLA).

Our records indicate we have not received your CLA. For legal reasons we need you to sign this before we can look at your contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

If you have recently signed the CLA, please wait at least one business day
before our records are updated.

You can check yourself to see if the CLA has been received.

Thanks again for your contribution, we look forward to reviewing it!

@dchevell dchevell mentioned this pull request Jan 11, 2019
Closed
@dchevell

dchevell commented Jan 12, 2019

Copy link
Copy Markdown
Contributor Author

Signed the PSF contributor agreement

pablogsal
pablogsal requested changes Jan 13, 2019
View reviewed changes
Comment thread Lib/concurrent/futures/process.py Outdated
# Liberate the resource as soon as possible, to avoid holding onto
# open files or shared memory that is not needed anymore
del call_item
del r

@pablogsal pablogsal Jan 13, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks incorrect as this will raise a NameError if the first call to call_item fails because the symbol was never bound.

@dchevell dchevell Jan 13, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pablogsal thanks for catching that. I've modified to handle this case and do nothing - although I'm not generally a fan of passing in except clauses it seems appropriate here.

@bedevere-bot

bedevere-bot commented Jan 13, 2019

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

pablogsal
pablogsal reviewed Jan 13, 2019
View reviewed changes
Comment thread Lib/concurrent/futures/process.py Outdated

@pablogsal pablogsal Jan 13, 2019

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You just need to delete the result in the else clause of the previous try block unless I am missing something.

@dchevell dchevell Jan 13, 2019

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point, thanks @pablogsal !

@dchevell

dchevell commented Jan 21, 2019

Copy link
Copy Markdown
Contributor Author

@pablogsal any other feedback on this one?

@dchevell

dchevell commented Feb 27, 2019

Copy link
Copy Markdown
Contributor Author

Hey @pablogsal, any other feedback or changes needed here?

@pablogsal

pablogsal commented Mar 3, 2019

Copy link
Copy Markdown
Member

Sorry for the delay. Please, ping me if I have not landed this this week :)

@dchevell

dchevell commented Mar 11, 2019

Copy link
Copy Markdown
Contributor Author

@pablogsal did you mean to ping you about merging, or did I misunderstand?

No need to apologise for the delay, appreciate you taking the time :)

@pablogsal
pablogsal merged commit 962bdea into python:master Mar 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pablogsal pablogsal pablogsal approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dchevell @the-knights-who-say-ni @bedevere-bot @pablogsal