Streaming pull shutdown hangs indefinitely on error in callback #394
Description
Environment details
- OS type and version: 4.19.121-linuxkit
- Python version: 3.8.9
- pip version: 21.0.1
google-cloud-pubsubversion: 2.4.1
Steps to reproduce
- Use the StreamingPull subscriber
- Throw an exception in the callback function
- Will in some cases hang indefinitely
When an exception occurs in the callback, the subscriber will hang indefinitely in some cases (about 1 in 5). It seems the queue returns None instead of queue.Empty here:
python-pubsub/google/cloud/pubsub_v1/subscriber/scheduler.py
Lines 142 to 147 in 5812018
This happens both with await_callbacks_on_shutdown=True and await_callbacks_on_shutdown=False.
Code example
from concurrent.futures import TimeoutError
from google.cloud.pubsub import SubscriberClient
project_id = "your-project-id"
subscription_id = "your-subscription-id"
subscriber = SubscriberClient()
subscription_path = subscriber.subscription_path(project_id, subscription_id)
def callback(message):
raise Exception()
message.ack()
streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
print(f"Listening for messages on {subscription_path}..\n")
with subscriber:
try:
streaming_pull_future.result()
except TimeoutError:
streaming_pull_future.cancel()Stack trace
Exception in thread Thread-OnRpcTerminated:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.8/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 569, in _shutdown
dropped_messages = self._scheduler.shutdown(
File "/usr/local/lib/python3.8/site-packages/google/cloud/pubsub_v1/subscriber/scheduler.py", line 145, in shutdown
dropped_messages.append(work_item.args[0])
AttributeError: 'NoneType' object has no attribute 'args'