parallelize 'for' loop in Python 3

Question

I am trying to do some analysis of the MODIS satellite data. My code primarily reads a lot of files (806) of the dimension 1200 by 1200 (806*1200*1200). It do it using a for loop and perform mathematical operations.

Following is the general way in which I read files.

mindex=np.zeros((1200,1200))
for i in range(1200):
    var1 = xray.open_dataset('filename.nc')['variable'][:,i,:].data
    for j in range(1200):
        var2 = var1[:,j]
        ## Mathematical Calculations to find var3[i,j]## 
        mindex[i,j] = var3[i,j]

Since its a lot of data to handle, the process is very slow and I was considering parallelizing it. I tried doing something with joblib, but I have not been able to do it.

I am unsure how to tackle this problem.

This is an easy application of multithreading. A tutorial for the standard python library for this can be viewed here: pymotw.com/3/threading — ritlew
– ritlew, Commented Jul 13, 2018 at 13:04
@ritlew multithreading? mutiprocessing would be more performant to do so since the mathematical calculations could not be multithreaded anyway no? — Mathieu
– Mathieu, Commented Jul 13, 2018 at 13:12
Well if there are 806 files, each thread could take a file from a pool and perform relevant calculations on it. That's assuming I understand the problem correctly — ritlew
– ritlew, Commented Jul 13, 2018 at 13:14
@ritlew The reason I am saying this, is that because of the GIL, each process can only run one thread at a time. If I recall correctly, multithreading is efficient only when threads are not going to perform pythonic operation (openCV for instance works well with multithreading). — Mathieu
– Mathieu, Commented Jul 13, 2018 at 13:25

Mathieu · Accepted Answer · 2019-09-03 20:16:05Z

13

My guess is that you want to work on several files at the same time. To do so, the best way (in my opinion) is to use multiprocessing. To use this, you need to define an elementary step, and it is already done in your code.

import numpy as np
import multiprocessing as mp
import os

def f(file):
    mindex=np.zeros((1200,1200))
    for i in range(1200):
        var1 = xray.open_dataset(file)['variable'][:,i,:].data
        for j in range(1200):
            var2 = var1[:,j]
            ## Mathematical Calculations to find var3[i,j]## 
            mindex[i,j] = var3[i,j]
    return (file, mindex)


if __name__ == '__main__':
    N= mp.cpu_count()

    files = os.scandir(folder)

    with mp.Pool(processes = N) as p:
        results = p.map(f, [file.name for file in files])

This should return a list of element results in which each element is a tuple with the file name and the mindex matrix. With this, you can work on multiple files at the same time. It is particularly efficient if the computation on each file is long.

edited Sep 3, 2019 at 20:16

answered Jul 13, 2018 at 13:21

Mathieu

5,8547 gold badges34 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Vincent Over a year ago

Note that multithreading sould work fine too if the computational part is done with numpy.

Mathieu Over a year ago

@Vincent True numpy is indeed switching to C.

Mathieu Over a year ago

@Vincent However, it might need more adaptation of the code especially to do array-wide operation to obtain mindex instead of this double for loop.

Vincent Over a year ago

Since most of the time is probably spent during the IO operations and the numpy functions, I would expect mp.pool.ThreadPool to achieve the same performance (maybe even better since mindex doesn't have to be serialized). This would require a benchmark though.

Collectives™ on Stack Overflow

parallelize 'for' loop in Python 3

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related