Skip to content

feat(voice): add Gemini Live ephemeral token support - #56888

Merged
kitlangton merged 3 commits into
devfrom
feat/voice-gemini-live
Oct 19, 2026
Merged

kitlangton merged 3 commits into
devfrom
feat/voice-gemini-live

Conversation

@kitlangton

@kitlangton kitlangton commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

Clients calling POST /api/experimental/voice/session can mint ephemeral voice sessions for OpenAI (gpt-live, gpt-realtime) and xAI (grok-voice), but requesting a Google Gemini Live model fails with Unsupported voice model. Mobile and web clients need short-lived, single-use Google ephemeral tokens so they can connect directly to Gemini Live over WebSocket without holding a raw Google API key.

What Changes

Adds the gemini-live engine (provider: "google") to Realtime and connects Voice.Service to Google API key credentials via Integration (google, excluding google-vertex ADC credentials).

Model family Engine Provider Transport
gemini-3.8-live (-extended-thinking, etc.) gemini-live google WebSocket
sequenceDiagram
    participant Client
    participant Server as OpenCode Server (Voice.Service)
    participant Google as Google Auth Tokens (v1beta/auth_tokens)
    participant Live as Gemini Live WS (BidiGenerateContentConstrained)

    Client->>Server: POST /api/experimental/voice/session { model, session, transport: { type: "websocket" } }
    Server->>Google: POST /v1beta/auth_tokens (x-goog-api-key, uses: 1, bidiGenerateContentSetup)
    Google-->>Server: { name: "auth_tokens/..." }
    Server-->>Client: { type: "websocket", url: "wss://...BidiGenerateContentConstrained", token, expiresAt }
    Client->>Live: WebSocket connect (?access_token=token)
Loading

Ephemeral Token Minting

Realtime.connect posts to https://generativelanguage.googleapis.com/v1beta/auth_tokens with the server's Google API key in x-goog-api-key:

  • uses: 1 (single-use token)
  • newSessionExpireTime: 60 seconds from mint time (expiresAt returned as Unix seconds matching this window)
  • expireTime: 30 minutes from mint time for the active session stream
  • bidiGenerateContentSetup: locks { ...session, model: "models/${model}" } into the token as required by BidiGenerateContentConstrained (verified against the AuthToken REST schema and googleapis/python-genai wire serializer; note that the curl snippet in the ephemeral tokens guide mistakenly shows the SDK config key liveConnectConstraints instead of the wire field bidiGenerateContentSetup)
  • Returns { type: "websocket", url: "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContentConstrained", token: data.name, expiresAt }, leaving ?access_token=<token> for the client to append when connecting

References:

Requesting webrtc transport for a Gemini Live model fails before network I/O with Voice.InvalidRequestError (HTTP 400). When no stored Google key or GEMINI_API_KEY / GOOGLE_GENERATIVE_AI_API_KEY / GOOGLE_API_KEY environment variable is configured, Voice.Service returns Voice.UnavailableError (service: "google").

Scope

Covers server-side ephemeral token minting and google credential lookup for POST /api/experimental/voice/session. Client WebSocket audio framing and tool bridging live in the client apps.

Verification

bun --cwd packages/ai typecheck
bun --cwd packages/ai test test/realtime.test.ts
bun --cwd packages/core typecheck
bun --cwd packages/core run script/test.ts test/voice.test.ts
bun --cwd packages/server test test/voice.test.ts
bun --cwd packages/client test test/promise.test.ts
  • packages/ai/test/realtime.test.ts: passed (engine routing for active Gemini Live families and non-Live exclusions, v1beta/auth_tokens request headers/body and response decoding with deterministic TestClock, WebRTC rejection before network I/O).
  • packages/core/test/voice.test.ts: passed (GEMINI_API_KEY lookup via Integration, missing Google key UnavailableError).
  • packages/server/test/voice.test.ts and packages/client/test/promise.test.ts: passed.
  • Unverified in automated tests: no live network call was made against https://generativelanguage.googleapis.com/v1beta/auth_tokens or the BidiGenerateContentConstrained WebSocket endpoint.

Added by the opencode bot (AI-generated).

Public API: POST /api/experimental/voice/session accepts gemini-3.8-live* models over websocket (backwards compatible)

@kitlangton
kitlangton force-pushed the feat/voice-gemini-live branch from f9628ee to 817c56e Compare October 19, 2026 17:54
@kitlangton
kitlangton marked this pull request as ready for review October 19, 2026 19:56

@opencode-agent opencode-agent Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I ran the ai, core and server voice/realtime tests and the ai and core type checks on the PR, and all pass. I also confirmed from source that gemini-3.8-live* now routes to the gemini-live engine; on the base it falls through to Unsupported voice model. The Google env vars named in the new error message match the google integration's env methods, and google-vertex credentials stay out because it is a separate integration. Not exercised: real calls to Google's auth_tokens and Live WebSocket endpoints.

Public API: additive only. POST /api/experimental/voice/session now accepts gemini-3.8-live* models with websocket transport and returns { type: "websocket", url, token, expiresAt }. Existing models and callers are unaffected.

No blocking concerns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant