Default deadline for publish does not match expectations #303
Description
The default total deadline for publish operations is supposed to be 60 seconds. However, it actually seems to be 120 seconds. It is easy to test this by starting up python and running the following with internet disabled:
from google.cloud import pubsub_v1
publisher = pubsub_v1.PublisherClient()
project_id = INSERT PROJECT ID
topic_id = INSERT TOPIC ID
topic_path = publisher.topic_path(project_id, topic_id)
data="HI"
future = publisher.publish(topic_path, data.encode("utf-8"))
future.result()
This should exit after 60 seconds. However, it actually exits after 120 seconds with an error as follows:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 73, in error_remapped_callable
return callable_(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/grpc/_channel.py", line 923, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1614602541.607973000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5397,"referenced_errors":[{"created":"@1614602516.771517000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/cloud/pubsub_v1/publisher/futures.py", line 52, in result
raise err
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/cloud/pubsub_v1/publisher/_batch/thread.py", line 263, in _commit
Failed to publish 1 messages.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 73, in error_remapped_callable
return callable_(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/grpc/_channel.py", line 923, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/grpc/_channel.py", line 826, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1614602541.607973000","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":5397,"referenced_errors":[{"created":"@1614602516.771517000","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/retry.py", line 184, in retry_target
return target()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
six.raise_from(exceptions.from_grpc_error(exc), exc)
File "<string>", line 3, in raise_from
google.api_core.exceptions.ServiceUnavailable: 503 failed to connect to all addresses
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/cloud/pubsub_v1/publisher/_batch/thread.py", line 263, in _commit
response = self._client.api.publish(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/pubsub_v1/services/publisher/client.py", line 582, in publish
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func
return retry_target(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/retry.py", line 199, in retry_target
six.raise_from(
File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 120.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x7f8596277af0>, topic: "projects/<my project>/topics/<my topic>"
messages {
data: "HI"
}
, metadata=[('x-goog-request-params', 'topic=projects/<my project>/topics/<my topic>'), ('x-goog-api-client', 'gl-python/3.9.1 grpc/1.36.0 gax/1.26.0 gccl/2.4.0')]), last exception: 503 failed to connect to all addresses
response = self._client.api.publish(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/pubsub_v1/services/publisher/client.py", line 582, in publish
response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
return wrapped_func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func
return retry_target(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/google/api_core/retry.py", line 199, in retry_target
six.raise_from(
File "<string>", line 3, in raise_from
google.api_core.exceptions.RetryError: Deadline of 120.0s exceeded while calling functools.partial(<function _wrap_unary_errors.<locals>.error_remapped_callable at 0x7f8596277af0>, topic: "projects/<my project>/topics/<my topic>"
messages {
data: "HI"
}
Where are the default retry settings coming from and what are they? Want to confirm that we are trying requests more than once by default.