Make consumer check threaded to prevent wedge state #198
Conversation
|
Again, tail.py would need this change as well. |
|
Added same functionality to tail_manager.py and rebased |
|
Why a thread and not a process? My experience with threads in python has - thus far - been lame. Have you tried this in product? How does it far against master? |
Make consumer check threaded to prevent wedge state
|
So the thread usage here is very small, which helps. It basically is used to keep a timer going, to check the consumer process and ensure that it's still running. I found that if the producer process is queuing more than the consumer can process and the consumer then dies, the producer can end up filling the queue and blocking, thus being unable to get the consumer restarted. This results in a wedged state and the only resolution is to restart the process. So having a threaded timer allows the producer to wedge, but still be able to get the consumer started back up. |
2 participants
This moves the consumer process check and creation out into a timed thread. It allows the producer to end up in a wedged state (attempting to push to a full queue causing a block) and still be able to start the consumer process if it dies for some reason instead of remaining in a frozen state.