Skip to content
BranchesTags

Repository files navigation

Enterprise Agentgateway Workshop

agentgateway-architecture.gif

Prerequisites

Before starting this workshop, you will need:

  • Solo.io Trial License Key: Enterprise Agentgateway requires a valid license key. You can obtain a free trial license by visiting Solo.io or contacting Solo.io sales.
  • Kubernetes cluster (version 1.29.4 - 1.33.3 or compatible)
  • kubectl CLI installed and configured
  • helm CLI installed

See System Requirements for detailed cluster sizing, version support, and resource recommendations.

Table of Contents

Installation

Start here. All other labs depend on these two.

OpenShift users: Use the OpenShift-specific versions instead:

001 — Install Enterprise Agentgateway (OCP)

002 — Set Up Monitoring Tools (OCP)

Air-gapped / private-registry users: Use the air-gap variant that mirrors all chart-managed images into a private registry:

001 — Install Enterprise Agentgateway (Air-Gap)

Routing

Inference

Security

Rate Limiting

Guardrails

Transformations

MCP (Model Context Protocol)

Agent Frameworks

Agent Harnesses

Identity & Delegation

Evaluations

Load Testing

Observability

Upgrades & Lifecycle

Strategies for upgrading Enterprise Agentgateway without dropping traffic.

Platform Engineering

Use Cases

  • Support Kubernetes Gateway API
  • Install Enterprise Agentgateway
  • Configure agentgateway for LLM, MCP, and A2A consumption
  • Unified access point for consumption of LLMs
    • LLM Providers supported in this repo:
      • OpenAI
      • AWS Bedrock (IAM credentials, API keys, and EKS IRSA)
      • Anthropic (Claude)
      • Azure OpenAI (API key and AKS Workload Identity)
      • Google Vertex AI (user auth and GCP service account)
      • Groq (and any other OpenAI-compatible provider, via a host / pathPrefix override)
    • OpenAI Embeddings support
    • AWS Bedrock Titan embeddings support
    • OpenAI Batches API support (asynchronous batch processing)
    • Streaming responses support for real-time token generation
    • OpenAI Audio API support (Text-to-Speech and Speech-to-Text)
    • OpenAI Video Generation support (Sora)
    • Claude Code CLI integration with full observability
    • Claude Desktop integration as an MCP client
    • CrewAI multi-agent workflow integration
    • LangChain multi-agent pipeline integration
  • Identity & Delegation
    • OBO (On-Behalf-Of) token exchange fundamentals (impersonation + delegation)
    • CrewAI agent with MCP tools secured by OBO delegation
    • Microsoft Entra ID On-Behalf-Of (OBO) token exchange
  • LLM API Key Management
    • API Key masking in logs
    • Virtual keys — per-user API keys with independent token budgets and budget isolation
  • Token-based metrics from LLM
  • LLM request/response metadata in Traces
  • Traffic Routing patterns (path, host, header, query parameter, request body)
  • Semantic routing: one virtual model name (auto_model) resolved to an economy, mid, or high tier model by prompt content, using vLLM Semantic Router as an ExtProc processor in the PreRouting phase
  • Inference routing to in-cluster LLMs via the Gateway API Inference Extension (InferencePool + llm-d Endpoint Picker)
  • Model Evaluations
  • Security & Access Control
    • Control access with org-specific API-key
    • Control access with JWT authentication
    • JWT-based RBAC (Role-Based Access Control)
    • Keep an agent harness authenticated with Authorization Code + PKCE from a public client (no client secret) — Claude Code's apiKeyHelper renews a per-user Auth0 token silently, so the provider key never leaves the cluster and offboarding is an IdP action
    • Frontend TLS termination
    • Frontend mTLS with client certificate validation
    • SNI (Server Name Indication) matching for multi-domain HTTPS
    • OPA authorization with custom Rego policies (ext-auth)
    • BYO gRPC external authorization (ext-authz) for LLM and MCP routes
    • Tunnel a backend connection (e.g. JWKS fetch) through a corporate forward proxy via BackendTunnel (HTTPS_PROXY-style CONNECT)
  • Prompt Guard & Content Moderation
    • Comprehensive built-in Prompt Guard (prompt injection, jailbreak, PII, secrets, harmful content, encoding evasion, and more)
    • External moderation guardrails (OpenAI moderation API)
    • Advanced Webhook Prompt Guard
  • Prompt Enrichment
  • Rate Limiting
    • Rate Limit on a per-request basis
    • Local token-based rate limiting
    • Global token-based rate limiting
  • Request/Response Transformations
    • Response transformations
    • Header enrichment for observability
  • MCP (Model Context Protocol)
    • Route to in-cluster MCP servers
    • Route to external/remote MCP servers through AgentGateway
    • Dynamic MCP backends via label selectors (scale targets without editing the backend)
    • Expose existing REST APIs as MCP tools from an OpenAPI spec (external public APIs and in-cluster services)
    • Federate multiple MCP servers behind one backend (tool-name prefixing, FailOpen, per-persona tool filtering)
    • Composite MCP tools that fan out to multiple MCP + HTTP backends and merge the responses — aggregation, sequential orchestration, and structured output
    • MCP tool modes — Search (get_tool / invoke_tool meta-tools) and Code (run_code in a sandboxed JS runtime)
    • Secure MCP servers with JWT auth
    • BYO gRPC external authorization (ext-authz) for MCP routes
    • Eager OAuth with a pre-registered upstream IdP (Auth0 and Okta) — gateway acts as the OAuth Authorization Server visible to MCP clients
    • Pre-issuance entitlement gating — gRPC ext_authz hook gates OAuth token issuance per user, redirects denied users to a configurable URL
    • Two-layer OAuth for a real SaaS API (Figma) — eager OAuth front door (Auth0 or Microsoft Entra ID) plus per-user downstream credential forwarding to a vendor-provided IdP via token-exchange elicitation
    • Tool-level access control
    • Per-tool rate limiting for MCP traffic
    • Integration with Claude Code CLI
  • Direct Response / Health Checks
    • Configure fixed responses without backend calls
  • Timeouts and Retries
    • Request timeout configuration
    • Retry policies on specific error codes (503, etc.)
    • Observing how timeouts and retries interact together
  • LLM Failover
    • Priority group failover between LLM providers
    • Health-based routing across multiple backends
    • Failover on rate limit errors (429)
    • Intra-priority-group failover with per-provider eviction and P2C load balancing
    • 5XX server-error failover via a CEL unhealthyCondition
  • Load Testing with k6
    • Performance testing with k6 load generator
    • LLM and MCP traffic load testing
    • Ramping and constant load patterns
    • Integration with Grafana and Prometheus metrics
  • Observability & Cost Management
    • Per-user / per-key LLM cost tracking and chargeback via access logs and PromQL
    • Production observability, alerting, and autoscaling guidance
  • Platform Engineering
    • Platform/developer separation of concerns via two Helm charts
    • Platform team owns the gateway, cost tiers, security baseline (JWT/WAF), observability, and URL space
    • App teams self-serve LLM/MCP endpoints under a delegated path prefix without being able to set traffic policies
    • Structural governance via route delegation (label + namespace + prefix contract) and a strict developer values.schema.json
    • Assign and re-tier teams with a one-line platform values change; enable JWT gateway-wide without changing any team release

Validated on

  • Kubernetes 1.29.4 - 1.33.3
  • Enterprise Agentgateway v2026.8.2

User Stories / Acceptance Criteria

As a platform operator, I want the AI Gateway to apply granular token quotas and rate limits to requests based on either an API key or a user/group identified in a JSON Web Token (JWT), so that I can control costs, ensure fair resource usage, and have the necessary metrics and logs to enable real-time monitoring and accurate chargeback.

This section is a comprehensive list of all the functionality and data requirements.

Flexible Identification

  • The AI Gateway must be able to authenticate requests using either a static API key or by validating a JWT.
  • The gateway can be configured to identify the request source using the API key itself, or by extracting specific user_id and group_id claims from the JWT payload.

Dynamic Quotas and Rate Limiting

  • The platform operator can define and apply token quotas and rate limits to individual API keys, specific users, or entire user groups.
  • When a limit is reached, the gateway must enforce it by preventing further requests and returning an appropriate error response (e.g., 429 Too Many Requests).

Granular Token Usage Tracking

  • The gateway must track and log the number of prompt and completion tokens for every request.
  • Each log record must be tagged with the relevant identifier from the request (either the api_key_id or the user_id and group_id from the JWT).

Comprehensive Logging for Troubleshooting & Auditing

  • The gateway must generate structured, machine-readable logs for every request.
  • These logs must include all relevant data points: a unique request_id, timestamp, http_status_code, total_tokens, and the specific identifier of the request source.
  • The log format should be designed for easy ingestion into a centralized logging platform for long-term storage and detailed queries.

Metrics for Real-time Monitoring & Analysis

  • The gateway must expose a /metrics endpoint that provides real-time, Prometheus-compatible metrics.
  • The metrics must include dimensions that correspond to the request identifiers (api_key_id, user_id, group_id) and key usage data (tokens_consumed_total).
  • This enables the operator to create real-time dashboards and configure automated alerts (e.g., "Alert me if the Marketing group's token usage exceeds 80% of their monthly quota").

Data for Chargeback & Reporting

  • The combined logs and metrics must provide a complete and auditable data set that can be used to generate reports for cost attribution.
  • The operator can easily query or export usage data aggregated by api_key_id, user_id, or group_id over any given time period.

Management Mechanisms

  • The platform operator can manually set, adjust, and reset quotas for any user, group, or API key.
  • The system can also be configured to perform automated, recurring quota resets (e.g., at the beginning of each calendar month).

Why This is Important

This functionality is crucial for managing an enterprise-scale AI Gateway and directly addresses critical business needs:

  • Financial Control: By setting and enforcing token quotas, the organization can prevent unexpected cost overruns and maintain predictable spending on AI services.
  • Operational Excellence: Real-time metrics and detailed logs provide the necessary visibility to monitor system health, troubleshoot issues quickly, and ensure the gateway is performing as expected.
  • Organizational Governance: The ability to track and attribute costs to specific teams or departments facilitates an accurate chargeback model, making business units accountable for their resource consumption and promoting efficient usage.
  • Fair Access: Quotas and rate limits prevent a small number of users or applications from monopolizing resources and ensure that the AI services remain available and performant for all teams.

About

Workshop instructions for Enterprise Agentgateway

Resources

Readme
Activity
Custom properties

Stars

16 stars

Watchers

2 watching

Forks

6 forks
Report repository

Releases

Packages

Contributors

Languages