Routing
Route through model aliases and routing rules.
An AI Gateway Platform Built on Envoy's Proven Data Plane
Production-grade circuit breaking, weighted routing, and built-in OpenTelemetry tracing for your LLM gateway. Change one base URL to get started.
Based on Envoy
The infrastructure that runs production traffic at hyperscale, applied to LLM requests
Built on the Agent Router project (formerly Envoy AI Gateway) at the Agentic AI Foundation (AAIF). Tetrate co-created the project with Bloomberg. See published performance benchmarks for MCP overhead, control-plane scaling, and methodology.
- A C++ data plane that handles millions of requests per second without becoming a bottleneck at scale.
- Production-grade circuit breaking and outlier detection built for service-to-service traffic, applied to model providers.
- Run gateways fully managed by Tetrate, or self-host gateways in any cloud, region, on-premises site, or edge location. One management plane controls every gateway.
- Built-in OpenTelemetry emission at the proxy layer. Every request is a trace span with zero instrumentation in agent code.
How the AI gateway evaluates, protects, and logs every LLM request
Every agent request passes through the same pipeline inside the AI gateway. Each stage runs in order and either returns a response or passes the request on. This pipeline is what teams mean by an LLM gateway: a single control point between your agents and model providers. The independent Broadcom/VMware test of the Agent Router project (then named Envoy AI Gateway) measured about 2 ms of data plane overhead per request.
Look up the inbound API key to resolve owner, project, and team. All downstream stages use this context for quota checks, routing, and log attribution.
Token-window rate limits run here. Buckets configured per tenant decrement from the token counts the gateway filter emits.
The AI gateway extracts the model from the request, matches an AIGatewayRoute rule, and selects a backend: Anthropic, Bedrock, Vertex, OpenAI, Azure, your BYOK credentials, or self-hosted.
Send to the selected provider. On retry-eligible failures, such as configurable status codes (429, 500-503), connect failures, or timeouts, Envoy retries with exponential backoff, then shifts to the next-priority provider in the rule. The agent gets the backup provider's response.
Captures team, agent, model, provider, token counts, latency, retry count, and status. The log feeds spend rollups, dashboards, and your configured sink without touching agent code.
What each AI gateway stage gets you in practice
Failover
Automatic recovery from provider failures.
Logging
Every request produces a structured log record written to your configured sink.
Token limits and cost budgets
Cap tokens inline and track spend against budgets.
Traffic splitting
Distribute traffic across multiple models.
Works with the AI stack you already run
Tetrate Agent Router Enterprise is provider-neutral, framework-agnostic, and OpenAI-compatible. It sits in front of your existing stack.
Model providers
Self-hosted
models
models
Agent frameworks
Observability
Any OTel-
compatible
stack
compatible
stack
Identity and access
Any OIDC/SAML
provider
provider
Any SSO provider
Compare alternatives
Evaluating AI gateways? Start with our evidence-based comparison hub, then read the head-to-head guides.
Frequently asked questions about AI gateways
An AI gateway sits between an organization's applications and its AI model providers, so every model request passes through one control point. At that point the gateway routes requests, fails over between providers, enforces token limits, and records a trace for each request. Agent Router Enterprise runs this gateway on Envoy and exposes the gateway as one OpenAI-compatible endpoint.
An AI gateway retries a failed request, then sends the request to the next provider configured for the same model. The calling agent gets its response from the backup provider. In Agent Router Enterprise, Envoy retries with exponential backoff on configurable status codes, connect failures, or timeouts before shifting the request to the next-priority provider.
AI gateway vendors use different pricing models. Some vendors charge a share of the token spend they route, and others license the gateway itself. Pricing for Agent Router Enterprise depends on your deployment, and detailed quotes come through a demo request.
Yes, when the gateway separates its data plane from its management plane. The data plane can then run in your own Kubernetes cluster, region, or on-premises site, so model traffic stays inside your network. Agent Router Enterprise manages all of its gateways from one Tetrate-hosted management plane, and you control which request data, such as prompts in logs, reaches that management plane.
Ready to start routing AI traffic?
Start a fast-track evaluation: first routed request under five minutes, nothing to install.