Repository navigation
feat(voice): add Gemini Live ephemeral token support - #56888
Conversation
f9628ee to
817c56e
Compare
There was a problem hiding this comment.
Looks good. I ran the ai, core and server voice/realtime tests and the ai and core type checks on the PR, and all pass. I also confirmed from source that gemini-3.8-live* now routes to the gemini-live engine; on the base it falls through to Unsupported voice model. The Google env vars named in the new error message match the google integration's env methods, and google-vertex credentials stay out because it is a separate integration. Not exercised: real calls to Google's auth_tokens and Live WebSocket endpoints.
Public API: additive only. POST /api/experimental/voice/session now accepts gemini-3.8-live* models with websocket transport and returns { type: "websocket", url, token, expiresAt }. Existing models and callers are unaffected.
No blocking concerns.
Why
Clients calling
POST /api/experimental/voice/sessioncan mint ephemeral voice sessions for OpenAI (gpt-live,gpt-realtime) and xAI (grok-voice), but requesting a Google Gemini Live model fails withUnsupported voice model. Mobile and web clients need short-lived, single-use Google ephemeral tokens so they can connect directly to Gemini Live over WebSocket without holding a raw Google API key.What Changes
Adds the
gemini-liveengine (provider: "google") toRealtimeand connectsVoice.Serviceto Google API key credentials viaIntegration(google, excludinggoogle-vertexADC credentials).gemini-3.8-live(-extended-thinking, etc.)gemini-livegooglesequenceDiagram participant Client participant Server as OpenCode Server (Voice.Service) participant Google as Google Auth Tokens (v1beta/auth_tokens) participant Live as Gemini Live WS (BidiGenerateContentConstrained) Client->>Server: POST /api/experimental/voice/session { model, session, transport: { type: "websocket" } } Server->>Google: POST /v1beta/auth_tokens (x-goog-api-key, uses: 1, bidiGenerateContentSetup) Google-->>Server: { name: "auth_tokens/..." } Server-->>Client: { type: "websocket", url: "wss://...BidiGenerateContentConstrained", token, expiresAt } Client->>Live: WebSocket connect (?access_token=token)Ephemeral Token Minting
Realtime.connectposts tohttps://generativelanguage.googleapis.com/v1beta/auth_tokenswith the server's Google API key inx-goog-api-key:uses: 1(single-use token)newSessionExpireTime: 60 seconds from mint time (expiresAtreturned as Unix seconds matching this window)expireTime: 30 minutes from mint time for the active session streambidiGenerateContentSetup: locks{ ...session, model: "models/${model}" }into the token as required byBidiGenerateContentConstrained(verified against theAuthTokenREST schema andgoogleapis/python-genaiwire serializer; note that the curl snippet in the ephemeral tokens guide mistakenly shows the SDK config keyliveConnectConstraintsinstead of the wire fieldbidiGenerateContentSetup){ type: "websocket", url: "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContentConstrained", token: data.name, expiresAt }, leaving?access_token=<token>for the client to append when connectingReferences:
AuthTokenREST reference (v1beta/auth_tokens)BidiGenerateContentConstrained)Requesting
webrtctransport for a Gemini Live model fails before network I/O withVoice.InvalidRequestError(HTTP 400). When no stored Google key orGEMINI_API_KEY/GOOGLE_GENERATIVE_AI_API_KEY/GOOGLE_API_KEYenvironment variable is configured,Voice.ServicereturnsVoice.UnavailableError(service: "google").Scope
Covers server-side ephemeral token minting and
googlecredential lookup forPOST /api/experimental/voice/session. Client WebSocket audio framing and tool bridging live in the client apps.Verification
packages/ai/test/realtime.test.ts: passed (engine routing for active Gemini Live families and non-Live exclusions,v1beta/auth_tokensrequest headers/body and response decoding with deterministicTestClock, WebRTC rejection before network I/O).packages/core/test/voice.test.ts: passed (GEMINI_API_KEYlookup viaIntegration, missing Google keyUnavailableError).packages/server/test/voice.test.tsandpackages/client/test/promise.test.ts: passed.https://generativelanguage.googleapis.com/v1beta/auth_tokensor theBidiGenerateContentConstrainedWebSocket endpoint.Added by the opencode bot (AI-generated).
Public API: POST /api/experimental/voice/session accepts gemini-3.8-live* models over websocket (backwards compatible)