Skip to content

feat(experimental): add bidi stream retry manager. #1632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
Pulkit0110 wants to merge 31 commits into
base: main
Choose a base branch
from
Open

feat(experimental): add bidi stream retry manager. #1632

Pulkit0110 wants to merge 31 commits into from
+1,251 −307

Conversation

@Pulkit0110
Copy link
Member

@Pulkit0110 Pulkit0110 commented Nov 25, 2025 •
edited
Loading

  • Add bidi stream retry manager. It will the main class that would be responsible for retries.
  • Integrate MRD with bidi stream retry manager and reads resumption strategy
  • Added conformance tests for all error scenarios and smart resumption for bidi reads
  • Refactored MRD's download_ranges() method, which will now use retry manager's execute() method to send the requests
  • Modified the logic to parse and handle the BidiReadObjectRedirectedError.
  • Modified cloud build configuration to also pull the latest changes of the current PR.
  • Formatted few more files using black formatter

@Pulkit0110 Pulkit0110 requested review from a team as code owners November 25, 2025 09:26
@product-auto-label product-auto-label bot added size: l Pull request size is large. api: storage Issues related to the googleapis/python-storage API. labels Nov 25, 2025
@gemini-code-assist
Copy link
Contributor

gemini-code-assist bot commented Nov 25, 2025

Summary of Changes

Hello @Pulkit0110, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a foundational component for enhancing the reliability of asynchronous bidirectional streaming operations within the google-cloud-storage library. By providing a dedicated retry manager, it enables robust handling of transient failures, ensuring that streaming operations can automatically recover and continue without manual intervention. This change is crucial for improving the overall stability and user experience of experimental asynchronous features.

Highlights

  • New Bidirectional Stream Retry Manager: Introduced _BidiStreamRetryManager, a new class responsible for managing retry logic for asynchronous bidirectional streaming operations. This manager integrates with a _BaseResumptionStrategy to handle state management during retries and uses google.api_core.retry.AsyncRetry for defining retry policies.
  • Robust Retry Mechanism: The _BidiStreamRetryManager's execute method orchestrates the retry loop, allowing for automatic re-attempting of streaming operations upon retriable exceptions. It supports resuming operations from a previous state and updating state based on responses, ensuring resilience against transient network issues or service unavailability.
  • Comprehensive Unit Tests: Added a new test file test_bidi_stream_retry_manager.py with extensive unit tests for the _BidiStreamRetryManager. These tests cover various scenarios, including successful execution, retries on initial and mid-stream failures, handling of non-retriable errors, and proper behavior when retry deadlines are exceeded.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist bot reviewed Nov 25, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a _BidiStreamRetryManager for handling retries in bidi streaming operations, along with comprehensive unit tests. The overall implementation is solid and well-tested. I have a few suggestions to improve type safety and maintainability in bidi_stream_retry_manager.py, such as adding a missing type hint, making another one more specific, and pointing out the use of an internal attribute from google-api-core which could be a future maintenance risk. Please see my detailed comments.

@Pulkit0110 Pulkit0110 force-pushed the bidi-reads-retry4 branch 2 times, most recently from a3795a3 to 4f1aa96 Compare December 1, 2025 04:40
@Pulkit0110 Pulkit0110 added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 1, 2025
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 1, 2025
@chandra-siri chandra-siri added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 3, 2025
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Dec 3, 2025
@product-auto-label product-auto-label bot added size: m Pull request size is medium. size: l Pull request size is large. and removed size: l Pull request size is large. size: m Pull request size is medium. labels Dec 4, 2025
@Pulkit0110 Pulkit0110 requested a review from a team as a code owner December 16, 2025 10:30
@product-auto-label product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Dec 16, 2025
@Pulkit0110 Pulkit0110 force-pushed the bidi-reads-retry4 branch 2 times, most recently from c8da9ac to bea02a5 Compare December 23, 2025 05:43
@chandra-siri
Copy link
Collaborator

chandra-siri commented Dec 24, 2025

/gemini review

gemini-code-assist[bot]
gemini-code-assist bot reviewed Dec 24, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a robust retry manager for bidi streams, which is a significant enhancement for experimental async downloads. The implementation is well-structured, using a strategy pattern to separate the generic retry loop from the read-specific resumption logic. The addition of conformance tests to validate the retry behavior under various fault-injection scenarios is excellent. My review includes a few suggestions to improve maintainability and logging practices, such as replacing print statements with the logging module and addressing code duplication in error handling.

chandra-siri
chandra-siri requested changes Dec 24, 2025
View reviewed changes
Copy link
Collaborator

@chandra-siri chandra-siri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • General comments
    • convert print into log statements.
    • reduce code duplicates
    • add comments where ever there's non-trivial logic
  • In the internal bug attach screenshots of perf results with and without this change.

@chandra-siri
Copy link
Collaborator

chandra-siri commented Dec 24, 2025

Since it's a large change, can you make the description a little detailed. - https://www.conventionalcommits.org/en/v1.0.0/#examples

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chandra-siri chandra-siri Awaiting requested review from chandra-siri

1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

Requested changes must be addressed to merge this pull request.

Assignees

@chandra-siri chandra-siri

Labels

api: storage Issues related to the googleapis/python-storage API. size: xl Pull request size is extra large.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Pulkit0110 @chandra-siri @yoshi-kokoro