-
Notifications
You must be signed in to change notification settings - Fork 214
Closed
Labels
api: pubsubIssues related to the googleapis/python-pubsub API.Issues related to the googleapis/python-pubsub API.
Description
Environment details
- OS type and version: Ubuntu 22.04
- Python version: 3.11
- pip version: 23.0.1
google-cloud-pubsubversion: 2.18.4
Steps to reproduce
- Run code sample indefinitely
- Sometime it shows Assertion error in
_on_responsefunction.
Code example
def on_subscribe(subscription, until=None):
"""Decorator factory that provides subscribed messages to function.
It handle decorated function as callback. So message should be acked/nacked
inside decorated function.
Args:
subscription (str): Subscription ID.
Should be `projects/{PROJECT_ID}/subscriptions/{SUBSCRIPTION_ID}`
until (datetime.datetime): This function will subscribe messages
published until this timestamp.
"""
def _callback_factory(func, finished, subscribe_until, **kwargs):
def _callback(message):
"""Callback function.
It sends signal if subscribed all messages.
"""
publish_time = datetime.fromtimestamp(
message.publish_time.timestamp())
if subscribe_until and publish_time <= subscribe_until:
return func(message, **kwargs)
if subscribe_until and not finished.is_set():
logging.info('Subscribed all messages published until %s',
subscribe_until)
finished.set()
message.nack()
return _callback
def _wrapper(func):
@functools.wraps(func)
def _inner_wrapper(**kwargs):
# Event variable that is triggered when all messages are subscribed
all_subscribed = Event()
callback = _callback_factory(func=func,
finished=all_subscribed,
subscribe_until=subscribe_until,
**kwargs)
# Ensure closing subscriber for memory leak prevention.
with pubsub_v1.SubscriberClient() as subscriber:
future = subscriber.subscribe(
subscription=subscription,
callback=callback,
await_callbacks_on_shutdown=True,
flow_control=pubsub_v1.types.FlowControl(max_messages=5000),
)
all_subscribed.wait(timeout=60)
# Wait until future is finished when it's cancelled.
# If it cancelled by timeout or keyboard interrupt, ignore it.
try:
future.cancel()
future.result(timeout=60)
except (KeyboardInterrupt, TimeoutError):
pass
except Exception as e:
logging.error("Error occurs during subscription to %s: %s",
subscription, str(e))
return _inner_wrapper
return _wrapper
@on_subscribe(subscription="SUBSCRIPTION")
def callback(message):
# Do something with messageStack trace
Traceback (most recent call last):
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/api_core/bidi.py", line 657, in _thread_main
self._on_response(response)
File "/layers/google.python.pip/pip/lib/python3.11/site-packages/google/cloud/pubsub_v1/subscriber/_protocol/streaming_pull_manager.py", line 1107, in _on_response
assert self._scheduler is not None
Explanation
It's because future.cancel() executes manager.close() which makes _scheduler as None and it makes _on_response raise AssertionError.
Maybe it has to be protected by threading lock somehow.
chase-peach
Metadata
Metadata
Assignees
Labels
api: pubsubIssues related to the googleapis/python-pubsub API.Issues related to the googleapis/python-pubsub API.