Python Multiprocessing and Serializing Data

Question

I am running a script on a school computer using the multiprocessing module. I am serializing the data frequently. It can be summarized by the code below:

import multiprocessing as mp
import time, pickle

def simulation(j):
    data = []
    for k in range(10):
        data.append(k)
        time.sleep(1)
        file = open('data%d.pkl'%j, 'wb')
        pickle.dump(data, file)
        file.close()
if __name__ == '__main__':
    processes = []
    processes.append(mp.Process(target = simulation, args = (1,) ))
    processes.append(mp.Process(target = simulation, args = (2,) ))
    for process in processes:
        process.start()
    for process in processes:
        process.join()

So when I actually run my code for many more simulations and what I imagine to be more intensive varied tasks, I get the following error: IOError: [Errno 5] Input/output error usually preceded by file.open(...) or file.close().

My questions:

How do I fix this error in my script?
What does this error mean for a python newcomer? References appreciated.

Some more notes about my procedure:

Instead of setting the multiprocess attribute daemon to be True, I use screen to run the script and then detach. This allows me also to disconnect without worrying about my script stopping.
This seemed to be a related question about printing using the subprocess module. I did not explicitly use daemon as I said, so not sure if this will help.
This usually happens after running for about a day and occurs on different processes at different times.

At the suggestion of a friend, I used the bash command nohup instead of screen to decrease the number of 'layers' between me and the kernel. He said roughly speaking this might increase the chances that this error doesn't crop up. Anyways, it still does... — Charlie
– Charlie, Commented Oct 3, 2014 at 1:44

johntellsall · Accepted Answer · 2014-10-07 15:28:59Z

3

Your program looks pretty good. In this case IOError just means "bad things happened." The entire set of simulated data became to large for the Python process, so it exited with the mysterious message.

A couple improvements in the following version:

Once some data has been produced, append it to a data file, then zap it from memory. The program should have roughly the same RAM use over time, rather than using up more and more, then crashing.
Conveniently, if a file is a concatenation of pickle objects, we can easily print out each one later for further examination. Example code shown.

Have fun!

source

import multiprocessing as mp
import glob, time, pickle, sys

def simulation(j):
    for k in range(10):
        datum = {'result': k}
        time.sleep(1)
        with open('data%d.pkl'%j, 'ab') as dataf:
            pickle.dump(datum, dataf)

def show():
    for datname in glob.glob('data*.pkl'):
        try:
            print '*'*8, datname
            with open(datname, 'rb') as datf:
                while True:
                    print pickle.load(datf)
        except EOFError:
            pass

def do_sim():
    processes = []
    processes.append(mp.Process(target = simulation, args = (1,) ))
    processes.append(mp.Process(target = simulation, args = (2,) ))
    for process in processes:
        process.start()
    for process in processes:
        process.join()

if __name__ == '__main__':
    if '--show' in sys.argv:
        show()
    else:
        do_sim()

output of "python ./msim.py --show"

******** data2.pkl
{'result': 0}
{'result': 1}
{'result': 2}
{'result': 3}
{'result': 4}
{'result': 5}
{'result': 6}
{'result': 7}
{'result': 8}
{'result': 9}
******** data1.pkl
{'result': 0}
{'result': 1}
{'result': 2}
{'result': 3}
{'result': 4}
{'result': 5}
{'result': 6}
{'result': 7}
{'result': 8}
{'result': 9}

edited Oct 7, 2014 at 15:28

answered Oct 4, 2014 at 3:13

johntellsall

15.2k4 gold badges48 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Charlie Over a year ago

THANK YOU very, very, VERY much for answering! I suspected some problems with memory. Still having trouble tracking this down, but I will check this again. I am using Matlab-like numpy implementations everywhere, so I have lots of code where I initialize a large array (~10^3), store data in individual array entries, and finally dump the array using pickle after each iteration. Use a dictionary to store different arrays by name. Don't know if I can alter all my code without a serious time investment. Any other fun workarounds, are appreciated, but again, so happy you answered.

Collectives™ on Stack Overflow

Python Multiprocessing and Serializing Data

1 Answer 1

source

output of "python ./msim.py --show"

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

source

output of "python ./msim.py --show"

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related