Skip to content

[Agent Engine] Deploy broken: base image has pydantic==2.13.4 + pydantic_core==2.47.0 simultaneously — incompatible versions cause SystemError at runtime #6041

@balakrishna-kudikala

Description

@balakrishna-kudikala

Environment

  • google-adk version: 1.0.0 (local: google-cloud-aiplatform==1.156.0)
  • Python: 3.12
  • Deployment target: Vertex AI Agent Engine (Reasoning Engine)
  • Region: us-central1
  • Last successful deploy: (approx. 2026-06-08)
  • First failing deploy: (2026-06-09) — same code, same requirements

Summary

The Agent Engine base image appears to have been updated with an inconsistent
pydantic state
: pydantic==2.13.4 and pydantic_core==2.47.0 are both
installed in /code/.venv, but pydantic 2.13.4 explicitly requires
pydantic_core==2.46.4. This causes a SystemError when the Agent Engine
runtime starts — before any user code runs.

pydantic 2.13.4 is the latest stable release on PyPI. No public pydantic
release requires pydantic_core==2.47.0. This is not a user-side configuration
issue — it is a broken base image.


Error (from Cloud Build logs, runtime startup)

File "/code/app/main.py", line 8, in
from app.settings import Settings
File "/code/app/settings.py", line 7, in
import pydantic_settings
File "/code/.venv/lib/python3.12/site-packages/pydantic_settings/init.py", line 2, in
from .main import BaseSettings, CliApp, SettingsConfigDict
File "/code/.venv/lib/python3.12/site-packages/pydantic_settings/main.py", line 13, in
from pydantic import ConfigDict
File "/code/.venv/lib/python3.12/site-packages/pydantic/init.py", line 8, in
_ensure_pydantic_core_version()
File "/code/.venv/lib/python3.12/site-packages/pydantic/version.py", line 94, in _ensure_pydantic_core_version
raise SystemError(
SystemError: The installed pydantic-core version (2.47.0) is incompatible
with the current pydantic version, which requires 2.46.4. If you encounter
this error, make sure that you haven't upgraded pydantic-core manually.

Note: /code/app/settings.py is Google's own Agent Engine runtime code,
not user code. The error occurs before the agent is even loaded.


Build log showing the conflicting install attempt

When requirements include pydantic (auto-appended by the SDK or explicit),
the build step also fails:

Step #2: Found existing installation: pydantic_core 2.47.0
Step #2: Uninstalling pydantic_core-2.47.0:
Step #2: ERROR: Could not install packages due to an OSError: [Errno 13]
Permission denied: '/code/.venv/lib/python3.12/site-packages/pydantic_core-2.47.0.dist-info/'

pip cannot downgrade pydantic_core from 2.47.0 to 2.46.4 because
/code/.venv is owned by root in the base image. This makes the broken
state unfixable from user requirements.txt.


Root cause

The Agent Engine base image contains both of these in /code/.venv:

Package Version Requires
pydantic 2.13.4 pydantic_core==2.46.4
pydantic_core 2.47.0 (no matching pydantic on PyPI)

These two versions are mutually incompatible. pydantic itself calls
_ensure_pydantic_core_version() at import time and raises SystemError.

pip install --upgrade pydantic on a clean environment returns 2.13.4
as LATEST — no pydantic version on public PyPI requires pydantic_core==2.47.0.


Steps to reproduce

  1. Deploy any ADK agent to Agent Engine (create or update):
    REQUIREMENTS = ["httpx>=0.27.0", "python-dotenv>=1.0.0"]
    agent_engines.create(app, requirements=REQUIREMENTS, extra_packages=["./"])
  2. SDK auto-appends pydantic==2.13.4 to requirements (detected from local venv)
  3. Build step: pip tries to downgrade pydantic_core 2.47.0 → 2.46.4 → Permission denied (pydantic_core is root-owned in base image)
  4. Even if pydantic install is skipped via environment markers, the Agent Engine runtime itself fails to start with SystemError because the base image already has the incompatible pydantic+pydantic_core pair

What was tried

Original requirements (httpx, python-dotenv only) : 400 Build failed — pydantic_core permission denied
Added google-cloud-aiplatform[adk]>=1.87.0 : Same — triggers pydantic chain
Added pydantic>=2.13.4 : pip picks minimum (2.13.4) → same conflict
Added pydantic-core>=2.47.0 : pip conflict — no pydantic satisfies both
Skip pydantic install via ; python_version < '3.0' marker : Build passes, runtime SystemError — base image is broken regardless

Expected behaviour

The Agent Engine base image should have a consistent pydantic + pydantic_core
pair. If the base image is updated to pydantic_core==2.47.0, the corresponding
pydantic release (e.g., 2.14.x) must also be present.

Workaround

None available until the base image is corrected or pydantic>=2.14.0
is released on PyPI with pydantic_core==2.47.0 support.

Metadata

Metadata

Assignees

Labels

agent engine[Component] This issue is related to Vertex AI Agent Enginerequest clarification[Status] The maintainer need clarification or more information from the author

Type

No fields configured for Bug.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions