Skip to main content
← Back to Release Notes
Latest

Envoy AI Gateway v1.1.x

Token counting, per-request credentials, stream idle timeout, MCP hostname routing, OpenTelemetry GenAI tracing, and HTTP CONNECT egress.

v1.1.0

August 21, 2026
Tokenize APICount TokensPer-Request CredentialsStream Idle TimeoutMCP HostnamesOTel GenAI TracingForward ProxyHelm Hardening
Envoy AI Gateway v1.1.0 is the first minor release on the stable 1.x API. It adds token counting across providers, per-request upstream credentials, stream idle timeout with failover, MCP hostname routing and CEL backend selection, optional OpenTelemetry GenAI tracing, and HTTP CONNECT egress. Upgrading from v1.0 needs no CRD migrations; the only operator-facing change is Helm's restricted controller security context.

✨ New Features​

Token Counting APIs

vLLM-compatible /tokenize across providers

Count tokens for a prompt without generating a completion. Chat messages and completion prompts use the vLLM tokenize request shape against vLLM, Vertex AI Gemini, GCP Anthropic, AWS Bedrock Converse, and AWS Anthropic. Native Anthropic clients can use /anthropic/v1/messages/count_tokens; Responses API clients can use /v1/responses/input_tokens (OpenAI and Azure OpenAI).

Anthropic-format /anthropic/v1/models

Claude clients that call Anthropic's model list endpoint now get the gateway's configured catalog in Anthropic's response shape, scoped by hostname the same way /v1/models is.

Per-Request Credentials & Egress

Per-request upstream credentials via credentialOverride

A trusted filter can supply the backend credential per request instead of sharing one static key in BackendSecurityPolicy. Source from Envoy dynamic metadata (preferred) or a request header the gateway strips before the backend. API keys, Anthropic, Azure, GCP, and AWS SigV4 are supported.

HTTP CONNECT forward proxy

GatewayConfig.spec.forwardProxy tunnels upstream LLM connections through an HTTP CONNECT proxy (host:port). Bedrock requests through PrivateLink now sign over the actual upstream host, so VPC endpoints no longer fail with a silent 403.

Streaming Resilience

Stream idle timeout with failover

AIGatewayRouteRule.streamIdleTimeout bounds how long Envoy waits without upstream bytes on a streaming response. If it fires before the first token, a retry policy can fail over to the next backend; mid-stream it returns 504. Pair with a BackendTrafficPolicy whose retry covers reset.

MCP Gateway

Hostname routing and backend selection

MCPRoute.spec.hostnames scopes an MCP endpoint to specific hosts. backendSelector restricts which backends a session fans out to via CEL rules evaluated at initialize (default Deny). Initialize now advertises merged backend capabilities, and the controller shares one Envoy Gateway Backend per namespace.

Observability

OpenTelemetry GenAI conventions and Grafana dashboard

Set AI_GATEWAY_TRACING_SEMCONV=gen_ai on the ext-proc to emit gen_ai.* span attributes. An example dashboard for gen_ai_* Prometheus metrics ships in examples/monitoring/grafana-dashboard.json. Controller and ext-proc logs accept --logFormat=json; MCP resource reads populate mcp_resource_uri in access-log metadata.

Provider Translation & API Compatibility

Structured output for Claude on Vertex AI

JSON-schema constrained decoding now works for Claude models on Vertex AI that advertise output_config. OpenAI reasoning_effort is forwarded to Bedrock as reasoning_config.

Responses API tool_search, unknown tools, and Codex inputs

The Responses API accepts the built-in tool_search tool and additional_tools, preserves unknown tool types, and accepts Codex-style agent input items.

Helm & Operations

Controller hardening and sharded config secrets

The chart can emit a PDB, honour topologySpreadConstraints, and attach extra podLabels. Filter configuration is split across multiple Secrets so large Gateway configs no longer hit Kubernetes' 1 MiB Secret size limit.

🔗 API Updates​

  • AIGatewayRouteRule.name: Optional unique name copied onto the generated HTTPRoute rule. route-not-found is reserved.
  • AIGatewayRouteRule.streamIdleTimeout: Optional Gateway API duration; unset means no per-try idle timeout.
  • BackendSecurityPolicy.spec.credentialOverride: Optional per-request credential source. Exactly one of fromRequestHeaders or fromDynamicMetadata. fallbackToConfigured defaults to true.
  • GatewayConfig.spec.forwardProxy: Optional HTTP CONNECT proxy. address is required (host:port).
  • MCPRoute.spec.hostnames: Optional list of hostnames (max 16).
  • MCPRoute.spec.backendSelector: Optional CEL selector evaluated at session initialize. defaultAction defaults to Deny.

📖 Upgrade Guidance​

Upgrading from v1.0 requires no CRD migrations. New fields are optional and backward compatible. Review the Helm security-context change if you override the controller image.

If the official controller image works for you, no action is needed. To restore empty security contexts:

controller:
  podSecurityContext: {}
  securityContext: {}

JSON logs: set controller.logFormat and extProc.logFormat to json. Enable a PDB with controller.podDisruptionBudget.enabled: true when you run more than one replica.

📦 Dependencies Versions​

Go 1.26.4

Continues on Go 1.26.4, same as v1.0.

Envoy Gateway v1.8.1

Built on Envoy Gateway v1.8.1 for proven data plane capabilities.

Envoy v1.38

Leveraging Envoy Proxy v1.38.1 for battle-tested networking.

Gateway API v1.5.1

Support for Gateway API v1.5.1 specifications.

Gateway API Inference Extension v1.0.2

Continued integration with Gateway API Inference Extension v1.0.2 for intelligent endpoint selection.

MCP Go SDK v1.7.0

Updated from v1.6.1 to v1.7.0 for the latest Model Context Protocol features.

⏩ Patch Releases​

🙏 Acknowledgements​

Thank you to everyone who made v1.1 possible

🔮 What's Next​

The roadmap is community-driven — join us and help shape it.

← Previous: v1.0.x SeriesAll Releases