PubSub crashes on timeout and can't recover (google.api_core error) #176
Description
Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
I'm going to do both, thanks.
Please run down the following list and make sure you've tried the usual "quick fixes":
- Search the issues already opened: https://github.com/googleapis/google-cloud-python/issues
- Check for answers on StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-python
Done and done.
If you are still having issues, please be sure to include as much information as possible:
Environment details
-
Specify the API at the beginning of the title (for example, "BigQuery: ...")
General, Core, and Other are also allowed as types -
OS type and version
Works on MacOS and debian linux locally and in GCB, does not work in debian linux on GKE -
Python version and virtual environment information:
3.7.6
- google-cloud- version:
google-api-core==1.22.0 google-auth==1.20.1 google-cloud-pubsub==1.7.0 googleapis-common-protos==1.52.0 grpc-google-iam-v1==0.12.3 grpcio==1.31.0
Steps to reproduce
- Run python code locally in MacOS, confirm that it works and tests pass.
- Run python code locally in docker container, confirm that it works and tests pass.
- Run python code on GCB as part of CICD process, confirm that tests pass and build succeeds.
- Deploy python code in docker container on GKE, watch it fail!
- Shell into the kubernetes pod, run python code there, watch it fail!
Code example
def async_pull(self) -> None:
"""
Asynchronously streams messages from pubsub and calls the callback() on them
"""
streaming_pull_future = self.subscriber.subscribe(
self.subscription_path, callback=self.callback
)
log.info(f"Listening for messages on {self.subscription_path}...")
with self.subscriber:
try:
streaming_pull_future.result(timeout=self.pubsub_timeout)
except TimeoutError:
streaming_pull_future.cancel()
logging.error(
f"Streaming_pull_future timed out while listening for messages on {self.subscription_path}"
)Stack trace
08-13 17:32:27 __main__: INFO Listening for messages on subscriber...
ERROR:root:Streaming_pull_future timed out while listening for messages on subscriber
08-13 17:32:32 __main__: INFO Listening for messages on subscriber...
ERROR:google.api_core.bidi:Thread-ConsumeBidirectionalStream caught unexpected exception Cannot invoke RPC: Channel closed! and will exit.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 637, in _thread_main
self._bidi_rpc.open()
File "/usr/local/lib/python3.7/site-packages/google/api_core/bidi.py", line 280, in open
call = self._start_rpc(iter(request_generator), metadata=self._rpc_metadata)
File "/usr/local/lib/python3.7/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client.py", line 1185, in streaming_pull
requests, retry=retry, timeout=timeout, metadata=metadata
File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
on_error=on_error,
File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/usr/local/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 144, in error_remapped_callable
result = callable_(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 1089, in __call__
operationses, event_handler, self._context)
File "/usr/local/lib/python3.7/site-packages/grpc/_channel.py", line 1183, in create
operationses_and_tags, context)
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 476, in grpc._cython.cygrpc.Channel.integrated_call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 294, in grpc._cython.cygrpc._integrated_call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 218, in grpc._cython.cygrpc._call
File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 258, in grpc._cython.cygrpc._call
ValueError: Cannot invoke RPC: Channel closed!
Making sure to follow these steps will guarantee the quickest resolution possible.
Thanks!