Parallelizing four nested loops in Python

Question

I have a fairly straightforward nested for loop that iterates over four arrays:

for a in a_grid:
    for b in b_grid:
        for c in c_grid:
            for d in d_grid:
                do_some_stuff(a,b,c,d)  # perform calculations and write to file

Maybe this isn't the most efficient way to perform calculations over a 4D grid to begin with. I know joblib is capable of parallelizing two nested for loops like this, but I'm having trouble generalizing it to four nested loops. Any ideas?

have you tried the obvious? Parallel(n_jobs=2)(delayed(do_some_stuff)(a, b, c, d) for a in a_grid for b in b_grid for c in c_grid for d in d_grid)? — Hamms
– Hamms, Commented Feb 2, 2017 at 22:55

Richard · Accepted Answer · 2017-02-02 23:04:47Z

38

I usually use code of this form:

#!/usr/bin/env python3
import itertools
import multiprocessing

#Generate values for each parameter
a = range(10)
b = range(10)
c = range(10)
d = range(10)

#Generate a list of tuples where each tuple is a combination of parameters.
#The list will contain all possible combinations of parameters.
paramlist = list(itertools.product(a,b,c,d))

#A function which will process a tuple of parameters
def func(params):
  a = params[0]
  b = params[1]
  c = params[2]
  d = params[3]
  return a*b*c*d

#Generate processes equal to the number of cores
pool = multiprocessing.Pool()

#Distribute the parameter sets evenly across the cores
res  = pool.map(func,paramlist)

answered Feb 2, 2017 at 23:04

Richard

62.8k40 gold badges199 silver badges277 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

godimedia Over a year ago

Is paramlist = [a,b,c,d] ?

Nesha25 Over a year ago

I ran this script, and paramlist looks like [(0, 0, 0, 0), (0, 0, 0, 1), (0, 0, 0, 2), (0, 0, 0, 3), (0, 0, 0, 4), (0, 0, 0, 5), (0, 0, 0, 6), (0, 0, 0, 7), (0, 0, 0, 8), (0, 0, 0, 9), (0, 0, 1, 0), (0, 0, 1, 1), (0, 0, 1, 2), (0, 0, 1, 3), (0, 0, 1, 4), (0, 0, 1, 5), (0, 0, 1, 6), (0, 0, 1, 7), (0, 0, 1, 8), (0, 0, 1, 9), (0, 0, 2, 0), (0, 0, 2, 1), (0, 0, 2, 2), (0, 0, 2, 3), (0, 0, 2, 4), ... (0, 9, 9, 6), (0, 9, 9, 7), (0, 9, 9, 8), (0, 9, 9, 9), ...]

user4815162342 · Accepted Answer · 2017-02-03 07:46:47Z

3

If you use a tool that makes it easy to parallelize two nested loops, but not four, you can use itertools.product to reduce four nested for loops into two:

from itertools import product

for a, b in product(a_grid, b_grid):
    for c, d in product(c_grid, d_grid):
        do_some_stuff(a, b, c, d)

edited Feb 3, 2017 at 7:46

answered Feb 2, 2017 at 23:00

user4815162342

159k22 gold badges351 silver badges419 bronze badges

2 Comments

Tedo Vrbanec Over a year ago

Significant acceleration, that's true. However, it is not parallelization but optimization. Still consuming one core.

user4815162342 Over a year ago

@TedoVrbanec By parallelization I referred to iteration over both sequences at once, not in the sense of using two CPUs. Also note that using itertools.product is no optimization either, it's just a different way of expressing the iteration.

jwd · Accepted Answer · 2017-02-02 22:58:43Z

2

The number of jobs is not related to the number of nested loops. In that other answer, it happened to be n_jobs=2 and 2 loops, but the two are completely unrelated.

Think of it this way: You have a bunch of function calls to make; in your case (unrolling the loops):

do_some_stuff(0,0,0,0)
do_some_stuff(0,0,0,1)
do_some_stuff(0,0,0,2)
do_some_stuff(0,0,1,0)
do_some_stuff(0,0,1,1)
do_some_stuff(0,0,1,2)
...

and you want to distribute those function calls across some number of jobs. You could use 2 jobs, or 10, or 100, it doesn't matter. Parallel takes care of distributing the work for you.

answered Feb 2, 2017 at 22:58

jwd

11.4k4 gold badges48 silver badges78 bronze badges

1 Comment

ylangylang Over a year ago

Right. I was mostly having trouble with structuring the code. I'm new to multiprocessing/joblib, so @Hamms's obvious solution somehow didn't come to mind. It does work though.

Collectives™ on Stack Overflow

Parallelizing four nested loops in Python

3 Answers 3

2 Comments

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related