1

I'm having trouble getting this example working. In serial this function works just fine, but when I attempt to run it in a multiprocessing.Pool it locks up and will not return a simple random integer. I'm specifically using the spawn context because I'm developing in a Windows environment.

import multiprocessing
from tqdm import tqdm


def test_parallel(min_rng: int):
    import random

    return random.randint(min_rng, 100)


def bootstrap_test_parallel(n_tasks: int = 10_000, min_rng: int = 3,
                            pool_size: int = max(1, multiprocessing.cpu_count() - 1)):
    results = [None for _ in range(n_tasks)]

    def log_result(value, ix):
        results[ix] = value

    if pool_size == 1:
        for ix in tqdm(range(n_tasks)):
            log_result(test_parallel(min_rng), ix)
    else:
        with multiprocessing.get_context("spawn").Pool(pool_size) as pool:
            for ix in tqdm(range(n_tasks)):
                log_result(pool.apply_async(test_parallel, args=(min_rng,)), ix=ix)

            for ix in tqdm(range(n_tasks)):
                results[ix] = results[ix].get()

    return results


if __name__ == "__main__":
    output_one = bootstrap_test_parallel(n_tasks=10, pool_size=1)  # runs fine
    print(output_one)
    output_two = bootstrap_test_parallel(n_tasks=10, pool_size=2)  # hangs indefinitely
    print(output_two)
3
  • This deadlock issue is a common pitfall when using multiprocessing. Pool and apply_async on Windows, especially with Python 3.8+, where the default start method is spawn, which has stricter requirements. Commented Jul 27, 2025 at 18:37
  • 1
    i think this is a problem with your IDE, if you run it from the terminal then it will work .... what IDE are you using ? some of them have special steps for running multiprocessing code. Commented Jul 27, 2025 at 22:07
  • 1
    It's common for multiprocessing to break when running inside a Jupyter Notebook. Jupyter does some hack to make it appear as if the executing cell is the __main__ module of the program. This trick does not get propagated to the child process, and so the child process ends up looking for test_parallel in the main module of the Jupyter Notebook Server. The solution is put test_parallel is a separate python module or to use cloudpickle. Commented Jul 28, 2025 at 0:35

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.