Skip to content

Bigtable client should retry ABORTED status codes similar to the hbase bigtable client #6205

@ykuida-E

Description

@ykuida-E

Is your feature request related to a problem? Please describe.

We've been seeing sporadic timeouts when reading from Bigtable

Exception in thread "main" java.lang.RuntimeException: com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Error while reading table 'projects/etsy-tanuki-prod/instances/tanuki/tables/search-denormalized-listings' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)

In our testing, we're attempting 20 concurrent bigtable reads of a large range of rows. At least one of these reads fail within ~3 minutes with the above error. As far as we can tell, there aren't any networking issues and we're processing the rows fairly quickly.

We noticed that the ABORTED status code isn't part of the default retryable codes. After adding it to the retryable codes, the issue went away.

I think adding the ABORTED status code to the default retryable codes would be nice. Currently the hbase client has it as one of the default retryable codes. https://github.com/googleapis/java-bigtable-hbase/pull/280/files#diff-b3f2d46de9814d2abefed2339765ba01R100

Describe the solution you'd like

I would like the default retryable codes to include the ABORTED code. (Or an understanding of why this is undesirable)

Describe alternatives you've considered

At the moment, I'm manually adding in the ABORTED code as a retryable code.

Metadata

Metadata

Labels

api: bigtableIssues related to the Bigtable API.type: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions