Implementing threads in python with threading module

Question

In a python program I need 2 threads based on Threading module. Thread#1 generates some data and put it in a buffer and thread#2 is supposed to process the data in the buffer.

So my pseudocode is like this: Thread 1:

Thread#1
while True:
   Generate_Some_Data()
   while flag==1:
      pass()
   Buffer_Address=Write_It_To_Buffer()
   flag=1


Thread#2
while True:
   while flag==0:
      pass()
   Process_Data(Buffer_Address)
   flag=0

(Let us assume that access to the variable "flag" is atomized with suitable locks.)

I know that Threading module is not concurrent. Essentially, this means that unless one of the threads does not block on some external condition (such as file-io or time.sleep), both threads will share the total process time regardless of the tasks. Therefore, according to my understanding, approximately half of the total processing time will be wasted on "while flag" loops in the above configuration.

So, here are my questions:

Am I right in my above anticipation/understanding of the Threading module? Is the half of the total process time wasted in "while flag" loops?
(If I am right) Is there anyway to get rid of "while flag" loops completely? I tried to find out another structure in which I can make threads sleep in lock.acquire() methods, however, I could not figure out a %100 safe way of doing so. (When one thread relases lock, there is no quarantee that the other thread will acquire it before the same thread acquire it again)

Taymon · Accepted Answer · 2012-03-07 07:03:15Z

3

It sounds like the Queue module is what you need.

This will give you a blocking FIFO queue. Whenever the consumer thread gets something from the queue, if there's nothing in there, it will block until something becomes available, giving the producer thread time to produce something.

If you're concerned about the producer thread hogging all the compute cycles, you can put a limit on the size of the queue. If the queue is full, and the producer thread has something to add, it will block until the consumer thread takes something out.

Here's some pseudocode:

import Queue
q = Queue.Queue()

# Producer thread
while True:
    data = generate_some_data()
    q.put(data)

# Consumer thread
while True:
    data = q.get()
    process_data(data)

answered Mar 7, 2012 at 7:03

Taymon

25.8k9 gold badges65 silver badges85 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

tantuni Over a year ago

I also have a follow up question. The python Queue's is an implementation of the semaphore concept which assumes that the consumer threads are identical. Assume I have 2 consumer threads which are supposed to process the data in the queue based on the data type. For instance, let us suppose that if data is of type "int" consumer 1 must process it. If it is "float" then consumer 2 must process it. However if it is "string" then neither consumer 1 nor 2 should process the data. Is there any programming paradigm that is useful for implementing such a problem?

Taymon Over a year ago

@user460153 I would simply make a separate queue for each consumer.

tantuni Over a year ago

I think "condition variable" is a better paradigm for this problem. All the consumer threads wait for the main thread to signal. After signaled by the main thread, each the consumer threads checks the data type and decide whether to process the data or continue to sleep.

Collectives™ on Stack Overflow

Implementing threads in python with threading module

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related