-
-
Notifications
You must be signed in to change notification settings - Fork 34k
Closed
Labels
3.13bugs and security fixesbugs and security fixes3.14bugs and security fixesbugs and security fixes3.15new features, bugs and security fixesnew features, bugs and security fixesdeferred-blockerstdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Description
Bug report
Bug description:
Below code demonstrates two problems that occur when
(a) worker process throws an exception, and
(b) shutdown(wait=False) is used
- When max_workers < number of submitted tasks, raises
exception TypeError: object of type 'NoneType' has no len() - When max_workers > number of submitted tasks, raises
exception TypeError: object of type 'NoneType' has no len()AND then program hangs
There is no problem when shutdown(wait=True) is used or if the worker processes exit cleanly.
import concurrent.futures
import multiprocessing as mp
import time
def task(n: int) -> int:
if n == 2:
raise Exception("Not gonna do it")
else:
time.sleep(n)
return n
def main() -> None:
# max_workers=2 results in raised exception and program exits
# max_workers=4 results in raised exception and program hangs
#
executor = concurrent.futures.ProcessPoolExecutor(
max_workers=2, mp_context=mp.get_context("forkserver"), max_tasks_per_child=1
)
f1 = executor.submit(task, 1)
f2 = executor.submit(task, 2)
f3 = executor.submit(task, 3)
result = 0
try:
result += f1.result()
result += f2.result()
result += f3.result()
print(f"Result = {result}")
except Exception as e:
print(f"Exception while getting result : {e}")
executor.shutdown(wait=False)
if __name__ == "__main__":
main()CPython versions tested on:
3.12
Operating systems tested on:
Linux, macOS
Linked PRs
- [3.12] gh-132969: Fix exception/hang shutdown(wait=False) and a task exited abnormally #133220
- gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally #133222
- [3.13] gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally (GH-133222) #135343
- [3.14] gh-132969: Fix error/hang when shutdown(wait=False) and task exited abnormally (GH-133222) #135344
- gh-132969: ACKS file update was missed in gh-133222 #136144
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
3.13bugs and security fixesbugs and security fixes3.14bugs and security fixesbugs and security fixes3.15new features, bugs and security fixesnew features, bugs and security fixesdeferred-blockerstdlibStandard Library Python modules in the Lib/ directoryStandard Library Python modules in the Lib/ directorytopic-multiprocessingtype-bugAn unexpected behavior, bug, or errorAn unexpected behavior, bug, or error
Projects
Status
Done