0

Good day!

I want to make an ExecutorService consumers for taking data from queue and working with it on server side. The idea is - I poll queue from time to time and if I see that it is not empty I start ExecutorService with N threads (lets say 5). Then I w8 while queue will be empty and shutdown threads. And all again - poll queue for data.... Is this alg ok? Or may be there are some ready implementations/frameworks for such task?

I found this implementation of ConcurrentQueue cunsumers :

public class ConcurrentQueueClient implements Runnable {

    private Queue<String> concurrentQueue;

    public ConcurrentQueueClient(Queue concurrentQueue) {
        this.concurrentQueue = concurrentQueue;
     }

    public void run() {
         boolean stopCondition = (concurrentQueue.size() == 0);

        while (!stopCondition) {
            for (int i = 0; i < concurrentQueue.size(); i++) {
                System.out.println("Client dequeue item "
                        + concurrentQueue.poll());

            }
            stopCondition = (concurrentQueue.size() == 0);
        }

        System.out.println("Client thread exiting...");
    }
 }

and testing it in such way :

 Queue<String> queue = new ConcurrentLinkedQueue<String>();
 ExecutorService consumers = null;
 while(true) {

if(queue.size() != 0) {
   consumers = Executors.newFixedThreadPool(100);
   for (int i = 0; i < 5; i++) {
        ConcurrentQueueClient client = new ConcurrentQueueClient(queue);
        consumers.execute(client);
   }
}

while (queue.size() != 0) {
       try {
           Thread.sleep(1500);
       } catch (InterruptedException e) {
           e.printStackTrace();
       }

}

 consumers.shutdown();
 try {
     consumers.awaitTermination(Long.MAX_VALUE, TimeUnit.NANOSECONDS);
 } catch (InterruptedException e) {
     e.printStackTrace();
 }


}
4
  • 1
    You should look into a BlockingQueue, mainly the put() and take(). Commented Feb 2, 2016 at 14:45
  • Ferrybig, why can't I use ConcurrentLinkedQueue ? Commented Feb 2, 2016 at 14:50
  • Berger, can you explain what do you mean? Commented Feb 2, 2016 at 14:50
  • 1
    @RedCollarPanda - As mentioned by @Ferrybig, try using BlockingQueue. With this you do not need to check for queue.size() != 0 but make use of the take() method which runs only when there is atleast one element in the queue. Commented Feb 2, 2016 at 15:53

1 Answer 1

1

Start over. Wrap your strings in callable or runnable, and queue those to the executor service.

If you have a finite set of data to process, then it's ok to have the main thread calling consumer.shutdown() and consumer.awaitTermination(...) as before, but no sleep loop. If you are going to process indefinitely from the queue, then no shutdown() until the service is.

You will face memory issues too if you don't have a limited blocking queue (nothing to block queue.put()). An ArrayBlockingQueue can be given to the executor service on creation (see ThreadPoolExecutor(...) )

Executor service's threads are doing that check (queue.take()) of the tasks queue by design. Try to avoid polling, it waste CPU. Always try to wait/notify (or await/signal) on conditions from reentrantlocks (which is all taken care of for you in the executor service code)

Sign up to request clarification or add additional context in comments.

2 Comments

Can you explane this? I need non-stop polling of queue. This polling will be in WebService method.
I must guess what you mean by that "polling will be in webservice method". If you mean that some servlet/rest api will go check the queue to consume all items parked in there, ok, but it's not really polling, you are trying to drain the queue really, and probably want to perform some hidden heavy processing you haven't revealed to us on multiple threads. So ok, that above is fine. Just dont have to loop on the sleep at all. awaitTermination will wait just fine.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.