gh-119592: gh-152967: Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits#152978
Conversation
…n a max_tasks_per_child worker exits Worker replacement went through the executor object: the manager thread read executor attributes that shutdown(wait=False) clears concurrently, and could not replace workers at all once the executor was garbage collected. A worker exiting at its max_tasks_per_child limit in those states left the remaining submitted work permanently unexecuted and hung interpreter exit; the racing case could crash the manager thread. Replace workers from the executor manager thread using its own state plus configuration read through the live executor weakref, which shutdown() never clears: - After shutdown(wait=False) with the executor still referenced, a replacement is spawned and the remaining work is executed as documented. - Once the executor has been garbage collected (pythongh-152967), or a replacement worker cannot be started and no workers remain, the remaining futures now fail with BrokenProcessPool instead of never resolving. - A new _force_shutting_down flag stops both spawn paths from starting workers that would escape terminate_workers()/kill_workers(). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
| self._join_executor_internals(broken=True) | ||
|
|
||
| def terminate_broken(self, cause): | ||
| def terminate_broken(self, cause, bpe_message=None): |
There was a problem hiding this comment.
While this might look like a public API change in the diff... it's on the _ExecutorManagerThread internal use only class. Fine to backport.
|
🤖 New build scheduled with the buildbot fleet by @gpshead for commit fd234c9 🤖 Results will be shown at: https://buildbot.python.org/all/#/grid?branch=refs%2Fpull%2F152978%2Fmerge If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again. |
|
Thanks @gpshead for the PR 🌮🎉.. I'm working now to backport this PR to: 3.14, 3.15. |
|
Sorry, @gpshead, I could not cleanly backport this to |
|
GH-153363 is a backport of this pull request to the 3.15 branch. |
…ted work when a max_tasks_per_child worker exits (GH-152978) (#153363) gh-119592: gh-152967: Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits (GH-152978) gh-119592: Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits Worker replacement went through the executor object: the manager thread read executor attributes that shutdown(wait=False) clears concurrently, and could not replace workers at all once the executor was garbage collected. A worker exiting at its max_tasks_per_child limit in those states left the remaining submitted work permanently unexecuted and hung interpreter exit; the racing case could crash the manager thread. Replace workers from the executor manager thread using its own state plus configuration read through the live executor weakref, which shutdown() never clears: - After shutdown(wait=False) with the executor still referenced, a replacement is spawned and the remaining work is executed as documented. - Once the executor has been garbage collected (gh-152967), or a replacement worker cannot be started and no workers remain, the remaining futures now fail with BrokenProcessPool instead of never resolving. - A new _force_shutting_down flag stops both spawn paths from starting workers that would escape terminate_workers()/kill_workers(). (cherry picked from commit 0c6422f) Reviewed-multiple-times-by: Gregory P. Smith Co-authored-by: Gregory P. Smith <68491+gpshead@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
Please don't forget about backports. |
Fix ProcessPoolExecutor stranding submitted work when a max_tasks_per_child worker exits. Worker replacement went through the executor object: the manager thread read executor attributes that shutdown(wait=False) clears concurrently, and could not replace workers at all once the executor was garbage collected. A worker exiting at its max_tasks_per_child limit in those states left the remaining submitted work permanently unexecuted and hung interpreter exit; the racing case could crash the manager thread.
Replace workers from the executor manager thread using its own state plus configuration read through the live executor weakref, which shutdown() never clears:
Drafted and investigated entirely by Claude Fable 5 based on the issues. edit: Looks to be in good shape. Undrafting. (I originally put this up as a draft to better iterate on review to see what shape this should take and how feasible backporting this further as a bugfix could be).