python multiprocessing JoinableQueue PicklingError

Question

Sorry...it seems that i asked a popular question, but i cannot find any helpful for my case from stackflow :P

so my code does the following things:

step 1. the parent process write task object into multiprocessing.JoinableQueue

step 2. child process(more than 1) read(get) the task object from the JoinableQueue and execute the task

my module structure is:

A.py

Class Task(object)
Class WorkerPool(object)
Class Worker(multiprocessing.Process)
- def run() # here the step 2 is executed
Class TestGroup()
- def loadTest() # here the step 1 above is executed, i.e. append the object of Task

What i understand is when mp.JoinableQueue is used, the objects appended should be pickable, i got the meaning of "the pickable" from https://docs.python.org/2/library/pickle.html#what-can-be-pickled-and-unpickled

my questions are: 1. does the object of Task is pickable in my case?

I got the error below when the code appends task objects into JoinableQueue:

File "/usr/lib/python2.6/multiprocessing/queues.py", line 242, in _feed

2014-06-23 03:18:43 INFO TestGroup: G1 End load test: object1

2014-06-23 03:18:43 INFO TestGroup: G1 End load test: object2

2014-06-23 03:18:43 INFO TestGroup: G1 End load test: object3 send(obj)

PicklingError: Can't pickle : attribute lookup pysphere.resources.VimService_services_types.DynamicData_Holder failed

What's the general usage of mp.JoinableQueue? in my case, i need to use join() and task_done()
When i choose to use Queue.Queue instead of mp.JoinableQueue, the pickerror is just gone, However, checking from the log, i found that all the child processes keep working on the first object of the Queue, what's the possible reason of this situation?

Jonathan Villemaire-Krajden · Accepted Answer · 2014-06-24 02:30:32Z

2

The multiprocessing module in Python starts multiple processes to run your tasks. Since processes do not share memory, they need to be able to communicate using serialized data. multiprocessing uses the pickle module to do the serialization, thus the requirement that the objects you are passing to the tasks be picklable.

1) Your task object seems to contain an instance from pysphere.resource.VimService_services_types. This is probably a reference to a system resource, such as an open file. This cannot be serialized or passed from one process to another, and therefore it causes the pickling error.

What you can do with mp.JoinableQueue is pass the arguments you need to the task, and have it start the service in the task itself so that it is local to that process.

For example:

queue = mp.JoinableQueue()
# not queue.put(task), since the new process will create the task
queue.put(task_args)

def f(task_args):
    task = Task(task_args)
    ...
    # you can't return the task, unless you've closed all non-serializable parts
    return task.result

process = Process(target=f, args=(queue,))
...

2) Queue.Queue is meant for threading. It uses shared memory and synchronization mechanisms to provide atomic operations. However, when you start a new process with multiprocessing, it copies the initial process, and so each child will work on the same queue objects, since the queue in memory has been copied for each process.

answered Jun 24, 2014 at 2:30

Jonathan Villemaire-Krajden

1,47811 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

3quanfeng Over a year ago

Thanks Jonathan, i put the case name(a string type) into the queue instead of the case object itself, no picklingerror reported anymore...

Collectives™ on Stack Overflow

python multiprocessing JoinableQueue PicklingError

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related