Skip to content

Bigtable client should retry ABORTED status codes similar to the hbase bigtable client #6205

Description

@ykuida-E

Is your feature request related to a problem? Please describe.

We've been seeing sporadic timeouts when reading from Bigtable

Exception in thread "main" java.lang.RuntimeException: com.google.api.gax.rpc.AbortedException: io.grpc.StatusRuntimeException: ABORTED: Error while reading table 'projects/etsy-tanuki-prod/instances/tanuki/tables/search-denormalized-listings' : Response was not consumed in time; terminating connection.(Possible causes: slow client data read or network problems)

In our testing, we're attempting 20 concurrent bigtable reads of a large range of rows. At least one of these reads fail within ~3 minutes with the above error. As far as we can tell, there aren't any networking issues and we're processing the rows fairly quickly.

We noticed that the ABORTED status code isn't part of the default retryable codes. After adding it to the retryable codes, the issue went away.

I think adding the ABORTED status code to the default retryable codes would be nice. Currently the hbase client has it as one of the default retryable codes. https://github.com/googleapis/java-bigtable-hbase/pull/280/files#diff-b3f2d46de9814d2abefed2339765ba01R100

Describe the solution you'd like

I would like the default retryable codes to include the ABORTED code. (Or an understanding of why this is undesirable)

Describe alternatives you've considered

At the moment, I'm manually adding in the ABORTED code as a retryable code.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

api: bigtableIssues related to the Bigtable API.type: questionRequest for information or clarification. Not an issue.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions