Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the ResumableStreamIterator to honor the configured maximum attempts and total timeout settings during streaming retries, preventing potential infinite retry loops. It also introduces comprehensive tests to verify this behavior. The review feedback is highly constructive, pointing out a potential sentinel value collision with System.nanoTime(), a numeric overflow risk in the timeout calculation, and a thread leak in the test suite due to an unclosed ScheduledThreadPoolExecutor.
- Normalize generated streaming retry defaults at the RPC boundary. - Extract retry handling and replace reflection with deterministic tests. - Document and test maxAttempts=1 as the way to disable streaming retries. - Cover StreamingRead attempt and timeout budgets with mock-server tests.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces bounded streaming retries for Spanner queries and reads by honoring RetrySettings (specifically maxAttempts and totalTimeout) in ResumableStreamIterator. It resets the retry budget when progress is made on the stream (i.e., when a new resume token is received). The default settings are normalized to preserve the historical unbounded resume behavior. Comprehensive unit and integration tests are added to verify these limits. Feedback on the changes highlights a potential issue where nextBackOffMillis returning -1L (indicating backoff has stopped) is not handled, which could lead to a negative sleep duration and an IllegalArgumentException instead of propagating the original SpannerException.
| long delayMillis = spannerException.getRetryDelayInMillis(); | ||
| if (delayMillis == -1L) { | ||
| if (this.backOff == null) { | ||
| this.backOff = newBackOff(); | ||
| } | ||
| delayMillis = nextBackOffMillis(this.backOff); | ||
| } |
There was a problem hiding this comment.
If nextBackOffMillis(this.backOff) returns BackOff.STOP (-1L), the backoff has stopped. Currently, if it returns -1L, totalTimeoutExceeded will receive -1L and return false (since Math.max(-1L, 0L) is 0L, which is less than the remaining timeout). This results in checkRetryBudgetAndGetDelay returning -1L, which is then passed to backoffSleep(context, -1L). Sleeping for a negative duration can cause an IllegalArgumentException (e.g., from Thread.sleep) instead of propagating the original SpannerException.
We should explicitly check if delayMillis is -1L after calling nextBackOffMillis and throw the original spannerException with the appropriate span annotations.
long delayMillis = spannerException.getRetryDelayInMillis();
if (delayMillis == -1L) {
if (this.backOff == null) {
this.backOff = newBackOff();
}
delayMillis = nextBackOffMillis(this.backOff);
if (delayMillis == -1L) {
span.addAnnotation(
"Stream broken. Not retrying because the backoff has stopped",
spannerException);
span.setStatus(spannerException);
throw spannerException;
}
}There was a problem hiding this comment.
I don't think this change is needed - the -1L/BackOff.STOP case is already handled safely, and the predicted
IllegalArgumentException can't actually occur here
2 participants
The resume loop in ResumableStreamIterator restarted a broken stream indefinitely for any retryable error, ignoring the maxAttempts and totalTimeout configured in the retry settings for ExecuteStreamingSql.
A streaming query could therefore retry forever when the server kept returning a retryable error, for example when a user configured DEADLINE_EXCEEDED as a retryable code and every attempt timed out.
The loop now counts consecutive failed attempts and stops retrying, rethrowing the last exception, when the configured maxAttempts is reached. It also enforces the configured totalTimeout as a wall-clock budget for a sequence of consecutive failed attempts, measured from the first failure of the sequence: a retry is only allowed when the
retry delay still fits in the remaining budget. This applies both to delays from the exponential backoff and to server-supplied retry delays (RetryInfo), which previously bypassed the backoff completely.
A totalTimeout of zero means that no time budget has been set, in which case only maxAttempts limits the retries, mirroring GAX.
Both limits only bind for custom retry settings: the default streaming retry settings do not set maxAttempts and keep the existing unbounded resume behavior. Progress on the stream resets both budgets, where progress means receiving a resume token that differs from the last seen token, so long-running streams that regularly make progress are
not terminated by an occasional transient error, while a stream that keeps returning the same token cannot reset the budget indefinitely.