Skip to content

Gemini model calls return 401 about 24h after start on Agent Runtime (Agent Identity)聽#7697

Open

Description

@JCorpe26

馃敶 Required Information

Describe the Bug:
An agent deployed to Vertex AI Agent Runtime with identity_type: AGENT_IDENTITY starts failing model calls with 401 UNAUTHENTICATED roughly 24 hours after the container starts. It keeps failing until the process is restarted. Under Agent Identity, access tokens are bound to a client certificate that rotates, and the long-lived genai.Client that Gemini.api_client caches appears to keep using the old one. Gemini offers no supported way to bound the client's age or rebuild it after an auth failure.

The one workaround, a Gemini subclass that evicts the cached client on a timer, was valid up to 2.9.x because the docstring recommended overriding api_client. In 2.10.0 api_client became a PerLoopCachedProperty stored under _per_loop_api_client, so evicting __dict__["api_client"] now silently does nothing. We have to maintain a private-attribute hack, which breaks when we update to the newer versions of the package.

Steps to Reproduce:

  1. Deploy any ADK agent using Gemini (default api_client) to Agent Runtime with identity_type: AGENT_IDENTITY.
  2. Send requests normally, then leave the engine running.
  3. About 24 hours after start, send another request.
  4. Model call fails with 401.

Expected Behavior:
Model calls keep working across certificate rotation. Failing that, Gemini should offer a supported way to bound client age (e.g. client_max_age) or rebuild on auth failure (e.g. an invalidate() that clears the per-loop cache).

Observed Behavior:
401 UNAUTHENTICATED from the model call about 24h in, until restart. With a subclass that rebuilds the client on a timer, the failures stop (the private attribute hack).

Environment Details:

  • ADK Library Version: 2.9.x
  • Desktop OS: Linux container on Vertex AI Agent Runt
  • Python Version: 3.14

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: gemini-3.5-flash

馃煛 Optional Information

Regression:
Partly. The 401 itself is not tied to an ADK version as far as I know. We did have a workaround, which no longer works in the newer versions of ADK.

Logs:
N/A

Additional Context:

  • Certificate rotation is the working explanation; I have not captured certificate notBefore dates, so it is not confirmed.
  • client_kwargs configures the client but does not bound its lifetime, so it is not a substitute.

Minimal Reproduction Code:
The 401 needs a deployed engine and about a day, it has no minimal repro code.

How often has this issue occurred?:

  • Always (100%) on a long-running engine.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions