1

I'm using Python multiprocessing pool imap_unordered.

In a generator, I'm filling my inputlist in fillInputList, sending this inputList to prepare_data_fill_arrays

prepare_data_fill_arrays takes a long time, it reads a file and fills the arrays.

When arrays are filled, I'm accumulating them in accumulated array

Both fillInputList and prepare_data_fill_arrays have print lines to show that they have entered to the functions.

However, none of them is written, but only:

  • Worker PID XXXX in accumulate

  • Worker PID XXXX in accumulate

These are written.

What can be the reason, it does not wait for prepare_data_fill_arrays to finish.

numofSimulations=10
chromNamesList=['chr1', 'chr2', 'chr3', 'chr4', 'chr5', 'chr6', 'chr7', 'chrX', 'chr8', 'chr9', 'chr10', 'chr11', 'chr12', 'chr13', 'chr14', 'chr15', 'chr16', 'chr17', 'chr18', 'chr20', 'chrY', 'chr19', 'chr22', 'chr21', 'chrM']
sim_nums = range(0, numofSimulations+1)
sim_num_chr_tuples=((sim_num,chrLong) for sim_num in sim_nums for chrLong in chromNamesList)

for arrayList in pool.imap_unordered(prepare_data_fill_arrays, (fillInputList(simNum,chrLong,x,y,z,a,b,c) for simNum,chrLong in sim_num_chr_tuples),chunksize=1):

    print('Worker pid %s in accumulate' %str(os.getpid()))
    array1 = arrayList[0]
    array2 = arrayList[1]
    accumulateArray(accumulatedArray1, array1)
    accumulateArray(accumulatedArray2, array2)
Improve this question
2
  • It's a process pool. It just starts the processes and returns. You should wait for all of them to complete what they're doing and then enter the for loop. Commented Nov 14, 2019 at 17:12
  • docs.python.org/2/library/multiprocessing.html How to wait for all of them to complete? In the python documentation for imap_unordered usage, it is called in for loop and there is no extra check mechanism. So what I understand is whenever a result is returned it enters the for loop. Commented Nov 14, 2019 at 17:13
Related questions
9
multiprocessing Pool.imap broken?
6
multiprocessing imap_unordered in python
5
Iteration over pool.imap_unordered
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.