fix: session resumption reconnection loop never iterates#5007
fix: session resumption reconnection loop never iterates#5007brucearctor wants to merge 5 commits intogoogle:mainfrom
Conversation
Three fixes for the run_live() reconnection loop: 1. Exception handlers in base_llm_flow.py now continue instead of raise when a session resumption handle is available. This covers both ConnectionClosed (from websockets) and APIError (from genai SDK wrapping ConnectionClosed). 2. send_history is skipped on reconnection — the server already has the session context via the resumption handle. 3. go_away messages from the server are now surfaced through gemini_llm_connection.py's receive() as LlmResponse.go_away, enabling proactive reconnection before server termination. Fixes: google#4996
Seven tests covering the exception handling and reconnection behavior of the outer while-True loop in run_live(): - test_reconnects_on_connection_closed_with_handle - test_reconnects_on_api_error_with_handle - test_raises_connection_closed_without_handle - test_raises_api_error_without_handle - test_raises_non_api_error_with_handle - test_skips_history_on_reconnect - test_sends_history_without_handle Uses a _LoopBreak sentinel and _make_connect_fn helper for deterministic loop termination in tests.
adk-bot
added
the
live
| logger.info( | ||
| 'Connection closed (%s), reconnecting with session handle.', e | ||
| ) | ||
| continue |
There was a problem hiding this comment.
This is great for brief network glitches. How do you think this would behave if the server was down for a few minutes?
There was a problem hiding this comment.
Good question! Right now the loop relies on the inherent connection timeout from llm.connect() to throttle reconnection attempts. For brief glitches that's sufficient, but you're right that a multi-minute outage would result in aggressive retries.
Makes me think:
- Add exponential backoff with jitter (e.g., 1s → 2s → 4s → ... capped at ~30s) between reconnection attempts
AND - Add a max retry count and raise after N failures
I'll add that to this PR. Or, would you suggest other?
rohityan
added
the
request clarification
| ) | ||
| async with llm.connect(llm_request) as llm_connection: | ||
| if llm_request.contents: | ||
| if llm_request.contents and not invocation_context.live_session_resumption_handle: |
There was a problem hiding this comment.
What happens if the resumption handle is rejected by the server?
There was a problem hiding this comment.
Great point. Looking at the Gemini Live API behavior: if the handle is rejected, the server sends back a session_resumption_update with resumable=False (or simply doesn't echo back a new_handle). It does not silently drop the context.
However, to be safe, we could:
Clear the handle on rejection: When we receive a session_resumption_update where resumable is False, clear live_session_resumption_handle so the next reconnection falls back to sending full history
Add a fallback: If the connection succeeds but no session_resumption_update arrives within a timeout, assume the handle was rejected and resend history
I think option (1) is already partially covered by the existing _receive_from_model logic that updates the handle from session_resumption_update.
Want me to verify the server behavior and add an explicit guard for the rejection case?
Add exponential backoff with jitter (1s base, 30s max) and a retry cap (10 attempts) to the run_live() reconnection loop. This prevents aggressive reconnection attempts during extended server outages. - Backoff delay: min(1s * 2^(attempt-1), 30s) + random(0,1) jitter - Max retries: 10 (configurable via MAX_RECONNECT_ATTEMPTS) - Attempt counter resets on successful connection - New tests: test_raises_after_max_retries_connection_closed, test_raises_after_max_retries_api_error Addresses reviewer feedback on PR google#5007.
Instead of re-raising the raw ConnectionClosedOK/APIError when max retries are exhausted, wrap it in a ConnectionError with a clear message and chain the original exception via 'from e'. This lets callers distinguish 'reconnection was attempted and exhausted' from a single unexpected disconnect.
3 participants
Summary
Fixes #4996 — the
run_live()reconnection loop was unreachable due to unconditional re-raising of exceptions, redundant history transmission on reconnection, and ignoredgoAwayserver messages.Changes
1. Exception handlers continue instead of raise (
base_llm_flow.py)ConnectionClosed/ConnectionClosedOKhandler nowcontinues thewhile Trueloop whenlive_session_resumption_handleis presentAPIErrorhandling (genai SDK wrapsConnectionClosedasAPIError) with the same continue-on-handle logic2. Skip
send_historyon reconnection (base_llm_flow.py)if llm_request.contents and not invocation_context.live_session_resumption_handle3. Surface
go_awaymessages (gemini_llm_connection.py+llm_response.py)GeminiLlmConnection.receive()now detectsmessage.go_awayand yields it asLlmResponse.go_awaygo_away: Optional[LiveServerGoAway]field toLlmResponseTesting
test_gemini_llm_connection.pytest_receive_go_away)test_run_live_reconnection.pytests/unittests/flows/llm_flows/(all)New reconnection tests cover:
ConnectionClosedOK/APIErrorwhen resumption handle existsAPIErrorexceptions always propagate, even with handlesend_historyskipped with handle, called without