-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Is your feature request related to a problem? Please describe.
We've been seeing sporadic timeouts when reading from Bigtable
Exception in thread "main" java.lang.RuntimeException: com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Error while reading table 'projects/etsy-tanuki-prod/instances/tanuki/tables/search-denormalized-listings' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)
In our testing, we're attempting 20 concurrent bigtable reads of a large range of rows. At least one of these reads fail within ~3 minutes with the above error. As far as we can tell, there aren't any networking issues and we're processing the rows fairly quickly.
We noticed that the ABORTED status code isn't part of the default retryable codes. After adding it to the retryable codes, the issue went away.
I think adding the ABORTED status code to the default retryable codes would be nice. Currently the hbase client has it as one of the default retryable codes. https://github.com/googleapis/java-bigtable-hbase/pull/280/files#diff-b3f2d46de9814d2abefed2339765ba01R100
Describe the solution you'd like
I would like the default retryable codes to include the ABORTED code. (Or an understanding of why this is undesirable)
Describe alternatives you've considered
At the moment, I'm manually adding in the ABORTED code as a retryable code.