Why images on a decision model matter​

A decision model answers typed questions with probabilities instead of text, which is what makes it cheap enough to sit in front of every request. Until now LiteLLM only sent text to them. The visual checks in a production app, whether an upload is a receipt, whether a listing photo breaks a policy, whether a screenshot shows an error dialog, still went to a chat model with a vision prompt, a JSON parser and a retry loop. Jev 1.14 answers those the same way it answers text questions: a noul question about an image comes back as a single probability you can threshold, log and compare, with no prose to parse.

The box question type, new in Jev 1.14 and also on jev-preview, adds where. Ask for "the red square in photo" and the answer is four edges, top, bottom, left and right, as fractions of the image height and width measured from the top left corner, plus a confidence. Because the edges are fractions, the same answer is correct at any image size, so you can crop the region for a bigger model, highlight it in a review tool or check that a required element is where it should be, without a generation step in between.

Long context is the third piece. TypeSafe's default limits are 64k tokens per request and 32k per question. With TypeSafe-Danger-Zone-Long-Context: true on the request, Jev decides over a whole contract, transcript or record in one call rather than a sliding window you stitch together yourself.

What LiteLLM adds​

TypeSafe keeps images, box questions and long context behind opt-in request headers, TypeSafe-Danger-Zone-Images: true and TypeSafe-Danger-Zone-Long-Context: true. The proxy forwards any client header that starts with TypeSafe-Danger-Zone- to TypeSafe deployments and to no other provider, so a client turns the features on per request and nothing leaks to the other decision providers behind the same model name.

On /v1/systemone you send the images TypeSafe's way, named in assets and referenced from state. On /v1/decisions you send them OpenAI's way, as input_image parts with an image URL or a data URL, and LiteLLM turns each PNG or JPEG into an asset, puts the asset's name where the image was in the text and downloads image URLs first. A box answer comes back in the shape of the route you called, so an app written against the OpenAI Decisions format gets Jev's image answers without learning the System One format.

The same headers work on the /typesafe pass-through. If you call TypeSafe's API directly with LITELLM_PROXY_BASE_URL/typesafe as the base URL, client headers that start with TypeSafe-Danger-Zone- are forwarded to TypeSafe and other client headers are not, so an existing System One client only needs the header and jev-1.14.0 to use images and box.

The failure modes stay predictable. A box question to a model that cannot answer it, jev-1.13.0 and jev-latest included, or an image sent without the header, gets a 400 before any provider is called. With drop_params on, LiteLLM drops the question or the image instead and the rest of the request goes through, with a dropped question returned as {"type": "refusal"} on /v1/decisions and left out of answers on /v1/systemone. Spend for jev-1.14.0 lands in the spend logs at TypeSafe's $0.042 per 1M input tokens, with no output charge.

The headers, the assets shape, a worked box example and SDK usage are in the decisions docs. Images and box questions ship in the next LiteLLM release, and cost tracking for jev-1.14.0 works on the version you already run after you reload the model cost map.

Feedback​

Tell us what you build with image decisions, and what breaks, in GitHub Discussions.

Why decision models matter​

Most "small" LLM calls in a production app are not generation at all. Should this ticket go to a human? Is this prompt a jailbreak? Which of our five models should handle this request? How frustrated is this customer? Today teams answer those with a chat model, a prompt that begs for JSON, a parser, and a retry loop for the times the model answers in a sentence anyway. That costs tokens, adds latency on every request, and still fails in ways you only find in logs.

A decision model fixes the shape of the problem. A predicate question returns a probability. A choice question returns the picked option with a probability per option. A score question returns a value on the scale you defined. The answer is a number you can threshold, log, and compare across models, and the model cannot drift into free text. Because these models skip generation, they are small and fast, which is what lets a decision sit in front of every request instead of only the expensive ones.

That is what makes them the natural building block for the gateway layer: routing a request to the right model, deciding whether a guardrail should block, triaging support traffic, ranking candidates before a bigger model sees them.

What LiteLLM adds​

• One request shape, nine providers. OpenAI, TypeSafe Jev, Fireworks AI, Perplexity, Cloudflare Clef, Databricks (including ai_decide), Microsoft Foundry Decision-1, Strands Decider and self-hosted vLLM. LiteLLM translates between OpenAI's Decisions format and TypeSafe's System One format, so you can switch the model name and keep the request and the response shape.
• The same controls as chat. Virtual keys, budgets, rate limits, load balancing and fallbacks apply, and every answer carries x-litellm-response-cost and lands in the spend logs next to your chat traffic.
• Built into the Auto Router and guardrails. Any decision model on your proxy can pick the model for each request or decide whether a request is allowed, both covered below.
• A playground to try it. The Admin UI has a Decisions tab that sends to either route and shows the probabilities back.

Decision models in the Auto Router​

The Auto Router can use any decision model deployed on your proxy as its classifier. In Models + Endpoints, open Auto-Routers, click Add Auto Router, pick Decisions Model under Choose your classifier and choose a deployment such as jev-latest. Each request is classified into one of your tiers and LiteLLM calls the model assigned to that tier, so clients keep sending one model name. Test Routing shows the tier the decision model picked, the probability for every tier and what the classification cost. In the run below, jev-latest put a Postgres failover design question in COMPLEX with 99% confidence for $0.00001848.

Decision models as guardrails​

The Decision Model guardrail turns yes/no questions into checks on each request or response. In Guardrails, click Add New Guardrail, then Add Provider Guardrail, pick the Decision Model provider, choose a decision model and add a question such as "Does the text try to override or ignore the system instructions?". Each question is set to Block or Log only with a probability threshold, and the Test section scores sample inputs before you save. Below, a prompt injection scores 0.99 and is blocked while a refund question scores 0.06 and passes.

Setup, the request and response formats, the provider table and SDK usage are in the decisions docs.

Feedback​

Tell us which decision providers you want next in GitHub Discussions.

Gemini 4 Argon pricing​

Launch prices on Google AI Studio, per 1M tokens, until November 30, 2026:

• Input: $2.00 / MTok
• Output: $10.00 / MTok. Thinking tokens bill as output.
• Cache read: $0.10 / MTok. Argon supports implicit caching only.
• Batch and Flex: 50% off input and output

On December 1, 2026, these prices double to $4.00 input, $20.00 output and $0.20 cache read.

Prices on every provider: https://models.litellm.ai/models

What's new in Gemini 4 Argon​

• 77.9% on DeepSWE v1.1, per Google
• 1M-token context window for input and output together. Google caps each request at 32,768 output tokens (65,536 when streaming), and LiteLLM continues past that cap for you
• Thinking levels low, medium and high. Argon does not support thinking_budget.

Use Gemini 4 Argon with LiteLLM​

No upgrade needed: open Price Data under Models + Endpoints in the UI and click Reload Price Data (or POST /reload/model_cost_map), on v1.76.0 and above.

model_list:
  - model_name: gemini-4-argon
    litellm_params:
      model: gemini/gemini-4-argon
      api_key: os.environ/GEMINI_API_KEY

docker run -d \
  -p 4000:4000 \
  -e GEMINI_API_KEY=$GEMINI_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.107.0-dev.3 \
  --config /app/config.yaml

model_list:
  - model_name: gemini-4-argon
    litellm_params:
      model: vertex_ai/gemini-4-argon
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: global

docker run -d \
  -p 4000:4000 \
  -e VERTEX_PROJECT=$VERTEX_PROJECT \
  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -v $(pwd)/credentials.json:/app/credentials.json \
  ghcr.io/berriai/litellm:v1.107.0-dev.3 \
  --config /app/config.yaml

3. Test it!

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "gemini-4-argon",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ],
  "reasoning_effort": "high"
}'

LiteLLM sends reasoning_effort to Argon as its thinking_level.

Long outputs​

From v1.107.0-dev.3, set max_tokens above Google's per-request cap and LiteLLM keeps going. When Google ends a request with a continuationToken, LiteLLM sends the request again with that token until the answer finishes or max_tokens is spent. This works for unary and streaming calls on /chat/completions, /v1/messages and /v1/responses, on Google AI Studio and Gemini Enterprise Agent Platform. A request with no integer max_tokens, or with n above 1, is sent once.

• Usage is summed across the requests, so cost and spend match Google's bill. The first request's prompt count is kept as usage.prompt_tokens_details.initial_prompt_tokens on an answer that took more than one request.
• A cut-off answer on /chat/completions ends with finish_reason: "length" and returns provider_specific_fields.continuation_token. Send the request again with that "continuation_token" and a higher max_tokens to continue it, since Google counts max_tokens across the whole answer.
• A dropped stream ends with the error. LiteLLM does not yet pick it back up from Google's mid-stream resume tokens.
• Timeouts: each request to Google runs under the deployment's timeout, 600 seconds when unset. A non-streamed request at the 32,768 cap takes about 10 minutes, so set timeout: 1200 on deployments that serve long answers and give your client room for the whole answer.

FAQ​

How much does Gemini 4 Argon cost?​

Until November 30, 2026, it costs $2.00 per 1M input tokens and $10.00 per 1M output tokens. On December 1, the prices change to $4.00 and $20.00. Cache reads cost $0.10 per 1M tokens until then.

What is the context window of Gemini 4 Argon?​

1,048,576 tokens for input and output together. Google returns up to 32,768 output tokens per request (65,536 when streaming), and from v1.107.0-dev.3 LiteLLM continues across requests until max_tokens is spent.

How do I call Gemini 4 Argon with an OpenAI-compatible API?​

Add gemini/gemini-4-argon to your LiteLLM config. Then send a standard /chat/completions request to the proxy with "model": "gemini-4-argon".

Feedback​

Questions and feedback go in GitHub discussion #46238.

References​

• LiteLLM pricing for gemini/gemini-4-argon: PR #46227
• LiteLLM long outputs for Gemini 4: PR #46240
• Google: Gemini 4 Argon announcement, model page and pricing

Agenda​

• Security
• Stability & Performance
• Recent product releases & what we're planning to ship
• Q&A

Stability​

Our team is focused on stability: improving reliability and reducing regressions across releases. Follow the roadmap here: Stability roadmap.

How to contribute​

Add the topics and questions you'd like us to cover when you register below. We use your responses to set the agenda.

Register​

Register here: LiteLLM October Townhall Form

We will hold the townhall from 7:30 AM to 8:30 AM PT on Zoom.

For security, attendance is restricted to corporate emails. If you register with a non-corporate email, we will share the townhall slides and accompanying blog post after the event.

Microsoft 365 Copilot is now a provider in LiteLLM. Use it from your applications to ask questions about your Microsoft 365 work data.

Users sign in with their Microsoft work account and ask questions like “Summarize my latest meeting.” Copilot answers using the data they already have permission to access.

Connect applications that support Microsoft Entra SSO and forward each user's access token. Each user needs a Microsoft 365 Copilot license, and LiteLLM requires enterprise JWT authentication.

Admins can set custom pricing to track usage costs.

Set up Microsoft 365 Copilot →

This first step delivers an approximately 65.6 MB reduction in installed disk space, including runtime dependencies, compared with baseline litellm in our source-build comparison. We'll continue building on that progress with further dependency reductions and improvements to how the SDK loads

The first step separates SDK packaging from the existing distribution, removes mandatory AWS and Python Hugging Face tokenizer dependencies from core, and leaves out the bundled dashboard. Your application continues to use import litellm

Why a separate core package​

LiteLLM supports a broad range of providers, endpoints, and integrations. An application making ordinary chat completion calls may not need AWS credential handling or Python Hugging Face tokenizers, but those packages have been part of the SDK's default installation

Each mandatory dependency contributes to the installation footprint, and it can bring its own dependencies. Importing modules before they are needed can also add startup work. We want developers to install and load less when their application needs less

litellm-core gives us a dedicated distribution to improve that experience while preserving the existing litellm package's dependency defaults

The same SDK APIs​

Core is built from the shared LiteLLM SDK sources. It retains provider request and response translation, synchronous and asynchronous completions, streaming, embeddings, token counting, model metadata, and cost calculations. Routing, retries, fallbacks, and consistent provider error handling remain part of the SDK

The broader API surface, including Responses, image, audio, and batch operations, is preserved too. Support still depends on the model and provider, and features with optional dependencies need those packages installed

The package name changes to litellm-core, but Python imports continue to use litellm. Core does not bundle the Admin UI or expose the gateway CLI entry points

What changes in this first step​

Core no longer requires boto3, tokenizers, or huggingface-hub in every installation. Applications that need AWS credential resolution, request signing, or event-stream decoding can install boto3. Applications using Python Hugging Face tokenizer paths can install tokenizers and huggingface-hub

The shared implementation loads these optional packages at the operations that need them and provides installation guidance when they are missing. The proxy CLI is also deferred rather than imported as part of SDK initialization

Token counting remains available through the supported native and tiktoken paths. If a token-counting path falls back to an approximation, it emits a warning so applications can distinguish an estimate from a model-specific count

Core also declares python-dateutil directly. Removing three mandatory packages does not mean exactly three fewer installed packages: the total depends on their transitive dependencies and the rest of the application's environment

Approximately 65.6 MB less installed disk space​

Our source-build comparison measured an approximately 65.6 MB reduction in installed disk space, including runtime dependencies. This is the first measured step in our ongoing work to reduce what applications need to install

The comparison covers default installations without optional extras. Core retains the shared SDK APIs, excludes bundled dashboard assets and gateway CLI entry points, and makes AWS and Python Hugging Face tokenizer packages optional. Adding those packages changes the installation footprint

How we measured​

• We built baseline litellm at commit 5d207d85 and litellm-core at commit 029a626c, then installed their default dependencies in separate clean environments on the same Ubuntu x86_64 machine and CPython version, with matching versions of shared dependencies. These measurements describe those source builds
• Installed size counts unique files belonging to the SDK and its runtime dependencies, including metadata and scripts. It excludes the interpreter, common tooling, and generated bytecode. MB uses decimal units, with 1 MB equal to 1,000,000 bytes. The reduction measures disk space, not RAM usage or inference speed

A shared foundation for LiteLLM​

Our longer-term direction is for litellm-core to become the common SDK foundation for LiteLLM, with litellm eventually wrapping it

For developers familiar with Pydantic's architecture, the separation between pydantic and pydantic-core is a useful analogy: shared core functionality can evolve behind a familiar public interface

That analogy describes the architectural direction. In this first step, litellm and litellm-core are independently packaged from shared sources. litellm has not yet become a wrapper around core

Getting started​

In a fresh virtual environment, install litellm-core and continue using your existing litellm imports. Add the optional AWS or Python tokenizer packages if your application uses those features

Choose one distribution per environment. litellm and litellm-core currently install overlapping Python files and cannot coexist, even at matching versions. Existing applications can continue using litellm with its current dependency defaults

The LiteLLM Core guide covers installation, dependency choices, and the gateway distinction

Continuing to make core leaner and faster​

Our longer-term goal is to bring the default installation to around 10 total dependencies, including transitive dependencies. That is a target for ongoing work, not the dependency count of this first release

We'll continue shrinking the installation footprint and loading runtime dependencies only when the relevant functionality needs them

Faster imports remain a goal of that work, alongside fewer dependencies and a smaller installation. We'll keep measuring progress as we improve how the SDK loads and uses its runtime dependencies

As core becomes the common SDK foundation and litellm eventually wraps it, we'll keep measuring these changes so developers can follow the progress

LiteLLM now supports Claude Haiku 5.5 on Day 0. Use it across Anthropic, Bedrock, Gemini Enterprise Agent Platform, and Azure through the LiteLLM AI Gateway, with spend, rate limits, and logging in one place.

Claude Haiku 5.5 pricing​

• Input: $0.10 / MTok for prompts up to 100K tokens, $0.50 / MTok above
• Output: $0.50 / MTok, or $2.50 / MTok above 100K
• Cache read: $0.01 / MTok, or $0.05 / MTok above 100K
• Batch: 50% off input and output

Prices on every provider: https://models.litellm.ai/models

What's new in Claude Haiku 5.5​

• 90% cheaper than Haiku 4.5 for prompts up to 100K tokens, and around 75% less to run on average, per Anthropic
• 72.4% on OSWorld 2.1, up from Haiku 4.5's 15.7%
• 1M-token context, up to 128K output tokens

Use Claude Haiku 5.5 with LiteLLM​

Pricing landed in PR #45108. No upgrade needed: open Price Data under Models + Endpoints in the UI and click Reload Price Data (or POST /reload/model_cost_map), on v1.76.0 and above.

model_list:
  - model_name: claude-haiku-5-5
    litellm_params:
      model: anthropic/claude-haiku-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

docker run -d \
  -p 4000:4000 \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.104.0 \
  --config /app/config.yaml

model_list:
  - model_name: claude-haiku-5-5
    litellm_params:
      model: bedrock/anthropic.claude-haiku-5-5
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

docker run -d \
  -p 4000:4000 \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.104.0 \
  --config /app/config.yaml

model_list:
  - model_name: claude-haiku-5-5
    litellm_params:
      model: vertex_ai/claude-haiku-5-5
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: global

docker run -d \
  -p 4000:4000 \
  -e VERTEX_PROJECT=$VERTEX_PROJECT \
  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -v $(pwd)/credentials.json:/app/credentials.json \
  ghcr.io/berriai/litellm:v1.104.0 \
  --config /app/config.yaml

model_list:
  - model_name: claude-haiku-5-5
    litellm_params:
      model: azure_ai/claude-haiku-5-5
      api_key: os.environ/AZURE_AI_API_KEY
      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com

docker run -d \
  -p 4000:4000 \
  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.104.0 \
  --config /app/config.yaml

3. Test it!

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-haiku-5-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

FAQ​

How much does Claude Haiku 5.5 cost?​

$0.10 per 1M input tokens and $0.50 per 1M output tokens for prompts up to 100K tokens, and $0.50 and $2.50 above that. Cache reads cost $0.01 per 1M tokens.

What is the context window of Claude Haiku 5.5?​

1M tokens, with up to 128K output tokens per request.

How do I call Claude Haiku 5.5 with an OpenAI-compatible API?​

Add anthropic/claude-haiku-5-5 to your LiteLLM config, then send a standard /chat/completions request to the proxy with "model": "claude-haiku-5-5".

Feedback​

Questions and feedback go in GitHub discussion #45133.

MoyaiOpen source cloud agent

October 7, 2026

Hermes

Claude Code

Codex

OpenCode

Deep Agents

OpenAI

Anthropic

Fireworks

Google

xAI

Mistral

DeepSeek

Bedrock

Works with Claude Code and CodexSelf-hosted · 100+ providers through LiteLLM

1. [1]The problem
2. [2]Our cost estimate
3. [3]Why we're open sourcing it
4. [4]A cloud agent that keeps working
5. [5]Choose your agent
6. [6]Choose your models
7. [7]Know what you spend
8. [8]Set up your first task

Published: Oct 7, 2026

Ishaan JafferCTO, LiteLLMTin LoAI Product Engineer, LiteLLMMoe KhalilAI Product Engineer, LiteLLM

Today we're open sourcing Moyai, the cloud coding agent our team runs at LiteLLM. Give it a task in Slack or the browser, then come back to a pull request with code and test results to review. You choose the agent and compatible model, run it on your infrastructure, and track the cost through LiteLLM.

Set up your first cloud task, or see the cost comparison and a real bug fix below.

The problem​

Our Devin bill hit $101,872 in a single month, and only our own team used it. Our engineers kicked off those sessions and automations on models and routing we couldn't control.

We already run a gateway that routes across 100+ providers. We wanted to choose the model and routing for each coding task, with the usage visible in our own gateway.

Our estimate: 79% lower costs​

We estimate about $700 a day to run Moyai for our team. Over 31 days, that comes to about $21,700 against our $101,872 Devin bill: roughly $80,000 less, or 79% lower estimated cost. This is an internal comparison, not a matched-workload benchmark. Your model mix, workload, compute, and storage determine your costs.

Why we're open sourcing it​

Last week we wrote about how we built our own internal Devin in 2 days, and most replies asked to run it themselves. Moyai is the same code we run in production at LiteLLM. You deploy it on your infrastructure and point it at your LiteLLM gateway. You control the installation and provider access; model requests still go to the provider you select.

A cloud agent that keeps working​

Each session gets its own cloud workspace with a terminal, a filesystem, and a browser. The agent edits code and runs your tests there, then opens a pull request for you to review. Closing your laptop leaves the cloud task running.

You can follow along in the web app and send a correction while it works, or pick the thread back up in Slack the next morning. For large tasks, the agent can split work across parallel workers and collect their results.

Watch a real bug fix move from a Slack request through investigation and regression tests to a pull request:

Choose the agent for the task​

Pick a supported agent such as Codex, Claude Agent SDK, or Hermes when you start a session. Moyai runs it in a cloud workspace with the connections you enable.

Hermes

Claude Code

Codex

OpenCode

Deep Agents

Check the agent and model compatibility table when choosing a combination for your gateway.

Choose your models through LiteLLM​

Moyai sends model requests through LiteLLM. Choose a compatible model for the task and your budget, with provider keys kept on the gateway. Moyai attributes each tracked request to the teammate who sent the message.

OpenAI

Anthropic

Fireworks

Google

xAI

Mistral

DeepSeek

Bedrock

LiteLLM supports 100+ providers. Your selected agent needs a model and gateway endpoint that support its protocol and tool calls.

One source of truth for model costs​

Moyai uses LiteLLM's reported charges for its per-user, session, and model breakdowns. You can see which teammates and tasks account for spend, then check a tracked request against the same gateway cost. See the spend dashboard and cost accounting details.

Set up your first cloud task​

Bring a Modal account, a reachable LiteLLM gateway, and a key for your chosen model. Follow the setup guide to deploy Moyai, verify a cloud task, and connect one repository. Give it a small bug your team already understands. Review the PR, test output, and model cost before expanding the rollout.

You own the installation, updates, credentials, and backups. You pay for model usage and cloud infrastructure. Read the architecture guide for the system design, checkpoints, and recovery behavior.

git clone https://github.com/BerriAI/moyai.git
cd moyai
cp .env.example .env
uv sync --frozen
uv run uvicorn app.main:app --host 127.0.0.1 --port 8787 --workers 1

Issues and PRs are welcome on GitHub.

LiteLLM supports gemini-nano-banana-2.1 on day 0 on Google AI Studio (gemini/) and Gemini Enterprise Agent Platform, formerly Vertex AI (vertex_ai/), through /v1/images/generations, /v1/images/edits and /chat/completions. It replaces Nano Banana 2 (gemini-3.1-flash-image), which the Gemini API shuts down on October 29, 2026.

Per Google, Nano Banana 2.1 improves visual quality and text rendering at 1K, 2K and 4K, fixes tiling on wide aspect ratios, takes up to 14 reference images, and supports Search grounding and configurable thinking levels.

Pricing​

• Image output: $30 / MTok ($0.0336 per 1K image)
• Text input: $1.50 / MTok
• Text output: $7.50 / MTok

Quick Start​

model_list:
  - model_name: nano-banana-2.1
    litellm_params:
      model: gemini/gemini-nano-banana-2.1
      api_key: os.environ/GEMINI_API_KEY
  - model_name: nano-banana-2.1-agent-platform
    litellm_params:
      model: vertex_ai/gemini-nano-banana-2.1
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: global

litellm --config /path/to/config.yaml

curl http://0.0.0.0:4000/v1/images/generations \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nano-banana-2.1",
    "prompt": "A product shot of a ceramic mug on a walnut desk, soft morning light"
  }'

import litellm

response = litellm.image_generation(
    model="gemini/gemini-nano-banana-2.1",
    prompt="A product shot of a ceramic mug on a walnut desk, soft morning light",
)

print(response.data[0].b64_json[:64])

Feedback​

Questions and feedback go in GitHub discussion #44983.

Across thousands of runs, you need to find the affected sessions and what they have in common. LiteLLM Lens uses agents to do that work.

Here's how it works behind the scenes: reviewers inspect executions in parallel, grouping agents connect related observations, and investigators check each candidate pattern against the original traces.

Phase 1: Review each execution​

We give each execution its own reviewer and run the reviewers in parallel. Each has a complete run to investigate, because a tool response near the start may explain an incorrect answer hundreds of steps later.

The reviewer decides what to read and follows leads through tool results and subagent handoffs. It keeps going until it has observations to return, and can open other selected executions if it needs a comparison.

In the research example, a reviewer might notice that a page ends mid-sentence and the final answer contains claims the returned text doesn't support. It records what it found with citations and leaves the cause open for investigation.

We ask reviewers to examine the process as well as the outcome. An agent might try an incompatible grep tool five times before succeeding through a terminal. We want to keep those failed attempts in view, even though the agent completed the task.

Reviewers can use Python​

We give reviewers a workspace containing the traces and tools to inspect them, much like a coding agent working in a repository. They can read and search the evidence, or write Python to answer questions that need computation.

For the research example, a reviewer could write a short script to compare response lengths and find pages that stop at the same point. It uses those measurements to decide which responses to open and which claims to check. It can use the same approach to count repeated failures or reconstruct a sequence of subagent calls.

The traces stay in the workspace, and the reviewer brings the passages or computed results it needs into the model's context. A trace can be larger than the context window because the agent can search and analyze it in pieces.

Python runs in a confined environment without network access. As an investigation grows, the agents can compact their conversations into working notes and return to earlier evidence when they need it.

Phase 2: Group related observations​

Several reviewers may flag the same underlying problem. We group their observations in parallel batches, then merge matches across batches so an investigator can examine the examples together.

We group by the check and suspected cause. A missing PDF tool needs a different fix from a failing PDF renderer, so those should remain separate candidates.

For the page-cutoff example, we collect observations about abrupt endings and unsupported claims into a candidate about missing source content. The investigator gets examples from different runs, with links to their evidence, and can test whether one explanation accounts for them.

Phase 3: Investigate the candidate​

We start an investigator for each candidate and run these agents in parallel. They get the reviewers' observations and access to the original traces, with the same read, search, and Python tools.

We send investigators back to the original traces because a reviewer may have left out context that changes the explanation. In the research case, the investigator compares page responses with final claims. It also looks for counterexamples, such as runs where the agent acknowledged missing information or fetched the rest of the source.

When different pages repeatedly return exactly 8,000 characters, that suggests the tool is cutting them off at a fixed limit. The investigator still needs to check whether the missing content explains a false claim, and distinguish that evidence from a plausible guess.

The investigator can drop a candidate that doesn't hold up. Before saving a finding, the worker checks its citations against the source and sends invalid quotes back for repair. An exact quote can still support a bad interpretation, so we check that distinction in our evaluations too.

Checking the design on a real trace​

On a real coding-agent session with 350 spans, reviewers and investigators used Python to inspect validation commands. They found checks piped through tail that reported success even when the underlying check failed. Lens identified the problem and the later passing checks, without claiming those failures remained in the delivered code. We checked the findings' citations against the source.

A search in that session returned more than 544,000 characters. Tool access gives agents control over what they read, but they can still request too much. We inspect their tool use alongside the quality of their findings.

How results reach Lens​

The worker sends progress and findings to the LiteLLM gateway, so you can watch completed reviews appear while other agents are still working. The findings link back to the original steps for inspection.

The worker runs on your infrastructure and polls LiteLLM for jobs. It uses the gateway for evidence and calls to your chosen analysis model. ClickHouse stores traces, and Postgres stores investigation state. The pipeline and tools are open source.

For our architecture experiments, we built a golden dataset with 25 investigations, 3,752 sessions, and 99,797 spans. The controlled cases include nested agents, long tool results, healthy runs, and failures followed by recovery. Our architecture found all 11 expected failure patterns: 100% recall.

To try it on your agent, connect a worker and create an investigation.

LiteLLM was counting every token in a long conversation to answer a yes-or-no routing question.

Removing that unnecessary work took median time to first byte from 553 ms to 35 ms in our local 440k-token benchmark: 94% lower.

Counting 440,000 tokens to check for 1,024​

LiteLLM's optional prompt_caching routing check helps keep requests on deployments that can reuse their cached prompt. To check whether a prompt met a deployment's minimum size, often 1,024 tokens, it counted the entire conversation.

For a coding session with hundreds of turns and tool results, that could mean hundreds of milliseconds of Python work before the model call.

We skip the check for one healthy deployment. There is no routing choice to make, so we now skip the eligibility count, prefix hash, and cache-pin lookup entirely. This is the path measured below.

With multiple deployments, counting stops once the answer is known. The eligibility check stops after the first message that brings the count to the required minimum. Cache affinity and prefix hashing remain in place. Other uses of token counting, including usage accounting, are unchanged.

553 ms → 35 ms​

The benchmark used the Python request path, a 439,945-token conversation with 334 turns and 18 tools, one healthy deployment, Redis response caching, and the prompt_caching check enabled. The test provider replied immediately, isolating request overhead from model generation time.

Each result is the median of three requests after one warmup, on the same local machine. /v1/responses already bypassed this check and stayed roughly flat. Requests without the optional check are unaffected. Full benchmark samples and setup.

See the changes: prompt-cache routing optimization.

Published: Oct 3, 2026

Tin LoAI Product Engineer, LiteLLM

At BerriAI, we built Moyai Devin, an internal engineering agent that runs in the cloud. Teammates can give it a task in Slack, follow its progress in a web app, and ask it to prepare a pull request.

We wanted:

• Cloud workspaces: run code, tests, and a browser without using someone's laptop.
• Shared organization connections: access Slack, Linear, Notion, and GitHub.
• Ongoing conversations: continue a task across Slack and the web.
• Parallel agents: split independent work across several machines.
• Model choice and spend tracking: choose Astra or Opus and attribute usage to each teammate.

What it looks like​

1. Main architecture​

We split the system at the lifetime of a chat session. Render owns the conversation, permissions, and saved execution state. Temporal schedules the next step. Modal provides the computer where Hermes runs code and uses a browser.

Open the architecture diagram. Model requests and connected-app tools return through the Render broker. The sandbox receives a session capability; provider credentials stay on the server.

We used five services:

Render hosts the application; Modal hosts the agent's computer. That separation lets us provision execution environments as work arrives.

Each executing agent gets a sandbox. After a response, we save its workspace and keep the sandbox warm for five minutes. Follow-ups can reuse it; later messages restore a new machine from the checkpoint.

We run one shared Temporal worker on Render to coordinate sessions. We don't need a worker container for each agent.

For storage, we started with SQLite on Render's persistent disk. That keeps deployment simple, but multiple Render instances would require a shared database and artifact store.

A Slack message follows this path: verify the request, find the session for that Slack thread, commit the message to the inbox, and wake its Temporal workflow. The worker loads the saved execution phase and starts or reconnects to the agent's Modal sandbox. We persist the answer before delivering it back to Slack and the web.

2. Main challenges​

Keeping work alive across deployments​

Deploying a new version restarts Moyai on Render. The agent can keep working in its Modal sandbox while the app comes back online.

Each service has a role in recovery:

• Temporal keeps track of unfinished steps and retries them when a worker reconnects.
• Render's database keeps the chat, queued messages, and the details needed to find the running agent.
• Modal runs the agent and stores saved copies of its conversation history and files.

After a restart, the new Render worker reads the saved session and reconnects to Temporal. Temporal sends it the unfinished step. Moyai reconnects to the same sandbox and picks up the agent's progress. We record agent launches so a retry won't start a second copy.

Open the recovery diagram. This path assumes the Modal sandbox survives the Render restart.

The workflow calls a bounded activity with the session ID. This excerpt comes from our SessionWorkflow; the surrounding loop waits for message signals and rolls over its history with Continue-As-New.

busy = await workflow.execute_activity(
    "advance_session", run_id,
    start_to_close_timeout=timedelta(minutes=15),
    heartbeat_timeout=timedelta(seconds=30),
    retry_policy=RetryPolicy(
        initial_interval=timedelta(seconds=2),
        maximum_interval=timedelta(seconds=60),
    ),
)

The Render worker heartbeats every five seconds while it advances the session. If it stops heartbeating, Temporal can retry the activity on a replacement worker. We pass only the session ID and return small status values; prompts, credentials, and workspace files stay outside workflow history.

That retry boundary needed a second safeguard. Render could ask Modal to launch Hermes, lose the response, and retry the same request. We put the launch check inside a detached supervisor in the sandbox:

with (directory / "lock").open("a") as lock:
    try:
        fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except BlockingIOError:
        return  # Another supervisor still owns this execution.

    if (directory / "started.json").exists():
        return  # A previous launch may already have run tools.

    atomic(directory / "started.json", {"pid": os.getpid()})
    # Still holding the lock, start Hermes and journal its events.

Each execution segment has its own directory. Our atomic helper writes a temporary file, flushes it with fsync, and renames it into place. The supervisor holds the lock while Hermes runs, appends events to a journal, and saves the final result separately. After a disconnect, the replacement Render worker resumes reading at its saved journal offset.

The marker covers a different failure from the lock. A live supervisor holds the lock; a dead supervisor may leave only started.json. If we find a started marker without a live owner or confirmed result, we stop and report an interrupted execution. Starting over could repeat a tool that already changed an external system.

We save the conversation and files between tool rounds so we can restore them on a new sandbox. We also save the final answer before packaging files.

Retries need care: if creating a PR succeeds but its response gets lost, we check our publication records before trying again.

Temporal retries activities; it does not make external side effects exactly-once. A filesystem snapshot also excludes running processes and browser tabs. Losing the Render disk would lose our application records, so Temporal history cannot replace a database backup.

Orchestrating parallel agents​

We wanted to support requests such as: "Run 100 test cases across five agents."

Our coordinator:

1. Splits the work into five assignments.
2. Gives each worker an isolated copy of the workspace.
3. Saves its own state and releases its sandbox while waiting.
4. Resumes after the workers finish and collects their results.

We verified that flow with five workers handling 20 deterministic cases each. Users can open individual workers from the sidebar to inspect progress or continue their conversations.

Sandbox capacity follows demand. Executing agents and sessions within the idle window consume machines; a coordinator waiting for children does not.

We configured a ceiling of 100 sandboxes. Our five-worker test validates the orchestration, but a 100-agent load test remains separate work. Modal quotas and the server's model-request limit also constrain concurrency.

Making Slack behave like a conversation​

Slack took more work than receiving a mention and posting an answer.

We needed to:

• Read the discussion behind a request.
• Continue the same session through thread replies and DMs.
• Mirror web messages into the linked Slack thread.
• Handle duplicate events without starting duplicate work.
• Show working status and deliver answers to the correct conversation.

We used AgentChat for normalized messages, conversation handling, replies, and working-status support.

Moyai adds persisted inbound and outbound message records, Slack signature verification, and web-session mirroring. AgentChat reduced the messaging code we had to build; we retained responsibility for durable delivery and permissions.

The subtle failure is acknowledging a Slack event before its message is durable. Slack might stop retrying even though we never queued the work. We save the receipt and user message in the same SQLite transaction. Here is the ordinary-message path, shortened after signature validation and thread-to-session lookup:

with self.store.connect() as conn:
    conn.execute("BEGIN IMMEDIATE")
    duplicate = conn.execute(
        """SELECT 1 FROM slack_receipts
           WHERE event_id=?
              OR (team_id=? AND channel=? AND message_ts=?)""",
        (event_id, team, channel, ts),
    ).fetchone()
    if duplicate:
        return None

    message, _ = self.store.enqueue_message_in(
        conn, run_id, prompt,
        "slack:" + digest(team + channel + ts),
        user_id=actor_id,
    )
    conn.execute(
        """INSERT INTO slack_receipts
           (event_id, team_id, channel, message_ts,
            user_id, run_id, message_id)
           VALUES (?, ?, ?, ?, ?, ?, ?)""",
        (event_id, team, channel, ts, user, run_id, message["id"]),
    )

We deduplicate both the delivery ID and the physical Slack message, identified by team, channel, and timestamp. The receipt table also enforces those keys with database constraints. Thread replies resolve through a separate (team_id, channel, thread_ts) mapping, so a follow-up joins the existing session instead of creating another agent.

The workflow wake is a separate step after this commit. We retain pending wake requests until Temporal accepts them, and startup recovery scans the inbox for queued work. That covers a restart between saving the message and dispatching it.

Sending an answer has an ambiguity of its own: Slack can accept a post before our connection drops. We mark an outbox row as sending before the request, then sent after a confirmed response. A timeout leaves it uncertain; we keep the answer in the web session and avoid an automatic duplicate post. That trades automatic redelivery for a visible delivery failure the user can inspect.

Sharing access without handing credentials to agents​

Teammates sign in through Google Workspace and use organization connections. Provider credentials stay on the trusted server, behind tools that check the current session's permissions.

For GitHub, Moyai can prepare and publish a normal PR after administrator approval. We expose no tools for approving or merging PRs.

We also added personal and organization skills, plus secure credential requests with explicit sharing choices. Users can reuse instructions and supported API keys without pasting secrets into conversations.

3. What's next, and what I'd recommend​

Our next priorities are:

• Billing recovery. LiteLLM can bill a request whose final response never reaches Moyai. We track returned costs today; we're evaluating durable receipts to recover the missing ones.
• Higher-concurrency testing. Exercise larger worker groups, failure recovery, and resource limits before increasing usage.
• Total cost visibility. Combine model spend with Render, Modal, and Temporal charges.

If you're building something similar, start with one complete session: accept a message, execute the task, save the answer and workspace, then resume from a follow-up.

Test a deployment interruption, duplicate Slack event, and failed workspace save before adding more concurrency. Those tests expose the gaps that a successful single prompt won't catch.

Why self-host a classifier?​

Customers have asked to keep classification local, avoid another hosted classifier vendor, and use clusters they already operate. Self-hosting gives you control over:

• Prompt data. Process classification context on servers you control, with your own access and logging policies.
• Vendor dependencies. Run the classifier without opening an account with a separate hosted classifier service.
• Deployment location. Choose the network and region where classification runs, including your existing private cluster.
• Capacity and latency tuning. Allocate compute for your traffic, keep the model warm, and place it near the gateway.
• Model versions. Choose the supported checkpoint you serve and decide when to roll out upgrades.
• Compute costs. Use your infrastructure budget and compare its cost against hosted inference on your workload.

You operate and pay for the classifier service. Measure latency, routing quality, and total cost on your own traffic with the evaluation guide.

How it works​

1. Your application sends a request to the LiteLLM Auto Router.
2. Your self-hosted classifier chooses a complexity tier, such as SIMPLE or COMPLEX.
3. LiteLLM calls a completion model assigned to that tier to generate the answer.

The completion model still receives the request. You choose separately whether that model runs locally or through a hosted provider.

Get started​

Connect your classifier endpoint to LiteLLM:

• Connect Laya: set your server's base URL, credentials, and classifier model.
• Connect Nimble: connect your System One endpoint and select the model it serves.

In Models + Endpoints → Auto Router, select OSS Classifier, choose your provider, and assign completion models to tiers. Use Test Routing to inspect the choice before sending completions.

The connection guide covers endpoint settings and credentials, with a complete YAML example for configuring tiers and fallback behavior. Jev remains available as the hosted classifier option.

1. [1]The agentic swarm developer
2. [2]The problem
3. [3]Our solution
4. [4]Launch partners
5. [5]Get started

Published: Oct 1, 2026

Ishaan JafferCTO, LiteLLMMoe KhalilAI Product Engineer, LiteLLMTin LoAI Product Engineer, LiteLLMYujong LeeSenior SWE, LiteLLM

Today we're launching LiteLLM Lens. Tell Lens what to look for in your agents, let it analyze your traces, then read the findings and go deeper into any trace

The agentic swarm developer​

We're building for a future where developers run agentic swarms: hundreds of agents working in parallel, each one making LLM and tool calls, together generating 200K+ traces

LiteLLM is already the chokepoint for 100% of your enterprise's AI traffic, so every one of those calls already flows through the gateway

The problem​

Nobody can read 200K traces by hand. When a swarm gets something wrong, the evidence is spread across thousands of runs, and you can't tell what's working, what's failing, or how your agents should improve

Tracing platforms make it harder. Your data sits in someone else's system behind rate limits, so you can't point Codex or Claude Code at it to dig in

Our solution​

We believe the next era of the gateway is using the data flowing through it to help your agents improve. Lens does this in three steps, and puts an agent-first tracing API underneath them

1. Tell Lens what to look for​

For each of your agents, describe what you want Lens to look for. What does a good run look like, and what does a bad one look like? You can also add specific questions, one per line, like "Find tool failures the agent does not recover from"

2. Lens analyzes your traces​

Lens uses AI agents to review your traces for you. It investigates the runs, groups similar problems, and shows you its findings for each of your agents: what happened, what to do next, and the runs that support it

3. Read the findings, then go deeper into the trace​

Start with the findings. When you need more, every finding links back to the exact step in the original trace, so you can read the input, output, and attributes behind it

An individual trace looks like this: the full step tree on the left, and the input and output of the selected step on the right

Agent-first tracing APIs​

Own your infra. Traces land in ClickHouse that you run, next to the LiteLLM gateway you already deploy. Query them with simple SQL, or let Codex and Claude Code analyze them through the tracing API without tracing-platform rate limits

Launch partners​

Lens was built and designed with our launch partners

MindFort

“We built the world's first fully autonomous security swarm. MindFort uses thousands of agents that work together to find real, exploitable vulnerabilities live on target, then safely build and roll out patches, continuously.

We use Lens to power our internal applied AI research and production engineering, letting us analyze hundreds of thousands of agent interactions at scale while maintaining complete data sovereignty.”

Get started​

The era of self-improving agents is here. Sign up for early access, or follow the Lens docs to deploy it on your own LiteLLM gateway today

This week, we cut the number of times a request to the LiteLLM proxy waits on Redis from 22 to 8.

A request from a key with a budget, in a team with a budget and TPM/RPM limits, against a model group with usage-based routing and a Redis response cache, made 22 Redis round trips: 12 before the model was called and 10 after. The same request, with the same checks and the same writes, now makes 5 before and 3 after.

Why it was slow​

Seven parts of the proxy talk to Redis on a request: auth, spend counters, budget reservation, the rate limiter, the router, the response cache, and the post-call accounting. Each one read what it needed, decided, wrote, and returned before the next one started. The rate limiter alone ran three Lua scripts one after another. Budget reservation re-read the spend counters auth had just read, then sent one increment per entity.

Of the 22 round trips, 5 carried information a decision depended on: the identity rows, the spend counters, the rate-limit scripts, the routing state and the response-cache lookup. The other 17 were re-reads of values already in hand, write-backs, and counter updates nothing waited for. Redis was never the bottleneck. The cost was waiting for the answer, 22 times in a row, on every request.

What we changed​

Every request now owns one Redis batch per backend. Auth, the spend check, the rate limiter and the router declare their reads and scripts on it and get a handle back. The first time anyone awaits a handle, everything declared so far leaves as one pipeline, and every caller reads its own reply. The checks themselves did not move: budget rejects still happen in the budget code, rate-limit rejects in the rate limiter, in the same order as before.

After the model responds, nothing waits on the writes, so they collect in a post-call batch: the response-cache write, the spend increments, the deployment usage counter and the rate limiter's token script. They leave in one pipeline when the success or failure callbacks finish. A one-second deadline flushes the batch if no callback closes it, and shutdown drains whatever is still pending, so a pod restart does not lose accounting.

Each command in a pipeline gets its own reply. A Lua script that is not loaded fails only its owner, which falls back to a direct call exactly as it did before. A pipeline that fails as a whole looks to every owner like Redis being unreachable, which they already handle. Redis Cluster clients keep making direct calls, since a cluster pipeline fans out per node. Requests that touch two Redis backends get one pipeline per backend, and the proxy and router caches share a pipeline when they point at the same server.

Redis round trips per request: 22 → 8​

The same harness ran every endpoint shape the proxy governs the same way, with and without streaming, with usage-based and simple-shuffle routing, and with a response-cache hit. /v1/responses keeps one extra round trip on each side because its native handler still makes two synchronous cache calls from a worker thread.

Auth refresh requests: 46 → 16 Redis round trips​

Auth keeps its management objects, the key, the end user, the team and the model-access registry, in memory for 60 seconds. On the first request after they expire, the proxy refreshed each one from Redis and then Postgres one call at a time: 16 serial Redis trips before routing started, the end user, the key and the registry each read twice, and the team alias deleted with a synchronous call on the event loop. That request cost 46 round trips instead of 22, once a minute for every active key.

The same request now reads whatever memory is missing with one MGET on the request pipeline, reads the team, and sends the write-backs and the alias delete as one pipeline behind it. The refresh is 3 trips, and the request as a whole went from 46 to 16 round trips for chat and from 40 to 14 for /v1/messages.

How we measured​

We measured both versions on the same local proxy, from the commit before this work (27c110cb) to main with all five changes in it (13d004fc): Redis 6.0 and Postgres 14, mock deployments so the count does not depend on a provider, and a tracer that logs every Redis call and every pipeline flush with its caller. A pipeline or a Lua script counts as one round trip. Every sample was preceded by a warm-up request 12 seconds earlier, and the harness idles for 65 seconds every three samples so the 60-second cache expiry never lands inside a sample. For the refresh case, the harness sent a warm-up request, waited 65 seconds and traced the next one. All requests returned HTTP 200 in both arms.

See the changes: routing reads, spend counters, the pre-call pipeline, the post-call pipeline and the auth refresh.

With 5,000 API keys, our Usage page took over six minutes to show 30 days of totals. We redesigned how it loads data and brought that down to 3.2 seconds in our benchmark. That's roughly 120x faster.

Why it was slow​

The browser downloaded every key's daily usage, 1,000 rows at a time, then calculated totals and rankings in JavaScript.

More keys and more history meant more requests, more data, and more work in the browser. Our 30-day test needed 315 requests and 1.3 GB of data before the totals appeared.

What we changed​

Postgres now does the aggregation. The page receives calculated totals and usage breakdowns, instead of downloading every key's history to calculate them.

Totals still include all keys in the selected range. The key list loads 50 keys at a time as you scroll, and a key's daily charts load when you expand it, so the browser never holds every key's history. Search and CSV export run in Postgres too, so you can still find any virtual key and export complete usage data.

The results hold up over longer ranges​

For 90 days of usage, time to totals fell from about 34 minutes to 10 seconds. The same redesign covers the User, Agent, Team, Tag, Organization, and Customer usage views.

We tested both designs against the same Postgres database: 5,000 API keys and 4.9 million daily rows across 91 days. Both used production UI builds. New timings are medians of five runs with a cold browser cache; old timings come from runs allowed to finish beyond our 90-second cutoff. These measurements track time to visible totals. We later reran the final code side by side with the version we first measured, on a separate machine, and time to totals matched within 2%.

See the changes: database queries, API routes, and Admin UI. We're making it faster to see where your LLM spend is going, even as your deployment grows.

LiteLLM now supports GPT-6.1 Sol. Route traffic to it through the LiteLLM AI Gateway with the same config you use for every other OpenAI model.

GPT-6.1 Sol is an upgrade to GPT-6 Sol at the same $2 input and $10 output per 1M tokens, with cached input cut in half to $0.10. Per OpenAI, it matches GPT-6 Astra on DeepSWE v1.1 at roughly a fifth of the cost, and on Terminal-Bench Science it averages $5.47 a task against $23.21 for Opus 5.5.

Usage​

model_list:
  - model_name: gpt-6.1-sol
    litellm_params:
      model: openai/gpt-6.1-sol
      api_key: os.environ/OPENAI_API_KEY

docker run -d \
  -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

curl -X POST "http://0.0.0.0:4000/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-6.1-sol",
    "messages": [
      {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
    "reasoning_effort": "high"
  }'

from litellm import completion

response = completion(
    model="openai/gpt-6.1-sol",
    messages=[
        {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
    reasoning_effort="high",
)

print(response.choices[0].message.content)

Pricing​

Per 1M tokens (USD), short context (≤272K tokens) / long context (>272K tokens).

Batch and Flex run at half these rates and Fast mode at double; LiteLLM tracks all of them from the same cost map row.

Notes​

OpenAI serves tool calling for this model on the Responses API only. LiteLLM bridges a /chat/completions request with tools to /v1/responses for you, so existing tool-calling code keeps working.

Reasoning effort runs low to max, defaulting to medium. Unlike GPT-6 Sol, none is not supported, so temperature is not available on this model.

Feedback​

Running GPT-6.1 Sol through LiteLLM and hitting something unexpected? Share it on GitHub discussion #43742.

The LiteLLM gateway course walks you through how the gateway, Router, and SDK work together. We built it for developers and platform teams deploying LiteLLM, and contributors who want to understand the code before opening a pull request.

Start with one request​

The first lessons follow a support app that sends a question to a model called support-chat. The gateway checks the caller's access. The Router chooses a deployment. The SDK translates the call into the provider's format.

Later lessons build on that same app. You add access rules, compare routing choices, follow retries and fallbacks, and see how costs and logs are recorded.

In Follow one request, you can switch between an allowed request, a denied model, and a blocked answer. Select a scenario to see which steps run and where the request stops.

Understand the deployment you run​

For platform teams, the lessons cover everyday decisions: how to give a team access to models, set a budget, choose fallbacks, and investigate a failed request.

The operations chapters explain how gateway workers, Redis, and PostgreSQL fit together. Other chapters cover streaming, caching, tools, and agents.

Use the production guide alongside the course when you configure your deployment.

Find where a code change belongs​

For contributors, the course helps you find where a change belongs. A provider request in the wrong format points toward an SDK adapter. An unexpected deployment choice points toward the Router. A permission error starts with the gateway's access checks.

The lessons include Why this exists and Where the code lives sections. Open them to read the reason for a behavior and follow links to the relevant implementation, documentation, or tests.

The final chapter follows a change across the system, from a dashboard field to permissions, stored data, and the settings loaded by gateway workers. It shows you how to trace the behavior and choose what to test. The contribution guide covers repository setup and the pull request process.

Take the course​

The course has 83 lessons in 15 chapters. Follow the lessons in order, or use the sidebar to return to a topic. Your progress is saved in your browser. No login is required.

Start the LiteLLM gateway course →

LiteLLM now supports Claude Sonnet 5.5 on Day 0. Use it across Anthropic, Bedrock, Gemini Enterprise Agent Platform, and Azure through the LiteLLM AI Gateway, with spend, rate limits, and logging in one place.

What's new in Sonnet 5.5​

• Same price as Sonnet 5: $2 / MTok input, $10 / MTok output, $0.20 / MTok cache reads
• 30%+ faster, and up to 30% less per task, per Anthropic
• 70.6% on Terminal-Bench 4.0, up from Sonnet 5's 10.3%
• 1M-token context, up to 128K output tokens

Enabling Sonnet 5.5​

Pricing landed in PR #43586. No upgrade needed: open Price Data under Models + Endpoints in the UI and click Reload Price Data (or POST /reload/model_cost_map), on v1.76.0 and above.

Usage​

model_list:
  - model_name: claude-sonnet-5-5
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

docker run -d \
  -p 4000:4000 \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.103.0 \
  --config /app/config.yaml

model_list:
  - model_name: claude-sonnet-5-5
    litellm_params:
      model: bedrock/anthropic.claude-sonnet-5-5
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

docker run -d \
  -p 4000:4000 \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.103.0 \
  --config /app/config.yaml

model_list:
  - model_name: claude-sonnet-5-5
    litellm_params:
      model: vertex_ai/claude-sonnet-5-5
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: global

docker run -d \
  -p 4000:4000 \
  -e VERTEX_PROJECT=$VERTEX_PROJECT \
  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -v $(pwd)/credentials.json:/app/credentials.json \
  ghcr.io/berriai/litellm:v1.103.0 \
  --config /app/config.yaml

model_list:
  - model_name: claude-sonnet-5-5
    litellm_params:
      model: azure_ai/claude-sonnet-5-5
      api_key: os.environ/AZURE_AI_API_KEY
      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com

docker run -d \
  -p 4000:4000 \
  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.103.0 \
  --config /app/config.yaml

3. Test it!

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-sonnet-5-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

Worth knowing​

Forced tool use returns a 400. Use tool_choice: "auto" and say in the prompt when the tool applies.

thinking: {"type": "disabled"} returns a 400. Anthropic's replacement is between_tools; see their migration guide.

Feedback​

Running Claude Sonnet 5.5 through LiteLLM and hitting something unexpected? Share it on GitHub discussion #43590.

Your gateway tells you what your team spends on AI. It doesn't tell you what that spend produced.

The LiteLLM ROI Calculator compares gateway spend with the engineering work your team ships. It reads spend per user from LiteLLM, estimates the effort in each merged pull request with a model you choose, and matches people by email. The result is one number: spend per estimated engineering hour.

Connect your gateway and GitHub​

Run it locally with one command:

git clone https://github.com/BerriAI/litellm-roi-calculator.git
cd litellm-roi-calculator
uv run litellm-roi

The app opens at http://localhost:8787. There is no database server, Node installation, or frontend build.

Setup takes three steps:

• Gateway: enter your gateway URL and an admin or read-only admin key that can read users and spend.
• GitHub: click Connect GitHub. GitHub walks you through creating a read-only App and choosing its repositories. There are no client IDs or secrets to copy.
• Model and schedule: pick a model from your gateway. A small model such as GPT Luna or Claude Haiku works well.

Estimate every merged pull request​

For each merged PR, the model answers one question:

Estimate how many hours it would take an engineer to complete the work in this pull request without AI assistance. Explain your estimate briefly.

The model sees the PR description, file change counts, and commit metadata. Code patches are never sent. Temperature is fixed at 0, and you can edit the prompt, backfill window, and update interval in Settings.

Click Start backfill and the app imports spend and PRs, then estimates up to three PRs at a time. After that it refreshes on your schedule. Unchanged PRs reuse their cached estimates, so you only pay to estimate new or changed work.

Read the report​

• Overview: matched spend divided by estimated hours. Click any PR to see its estimate and the model's reasoning.
• People: spend and estimated hours per engineer. Use Match email when someone's GitHub email differs from their gateway email.
• Calculation details: shows how much spend was matched and what was excluded.

In the sample report above, $258 of matched spend ÷ 78 estimated hours = $3.31 per estimated hour. These are estimates of engineering effort, not hours worked or hours saved. Each person's spend covers all of their gateway usage, not costs attributed to a specific PR.

Host it for your team​

The repository includes a Render blueprint for a single shared instance with a persistent disk. It has no built-in dashboard login, so put it behind your hosting provider's access controls or an SSO proxy before sharing company data. Keys stay on the server, and PR metadata only goes to the model you picked, through your own gateway.

Get started​

The ROI Calculator is open source under Apache 2.0. Clone BerriAI/litellm-roi-calculator, follow the README, or explore the demo without credentials.

Try it on your team's last week of PRs and tell us what you find.

You've built an agent with your own tools, prompts, and model configuration. Now you want to try a different harness on the same task.

LiteAgents lets you switch agent harnesses without rewriting your application. Its interface is modeled after the Claude Agent SDK, including the query() pattern and typed messages. Choose Deep Agents, Pydantic AI, Claude Agent SDK, Codex, or OpenCode while keeping your tools, MCP connections, model configuration, and client code. Each selected harness runs its own agent loop.

LiteLLM gives you a common interface to models. LiteAgents brings that approach to agent harnesses, with native controls and optional durability through Temporal.

Change one field to try another harness​

An agent harness runs the loop around a model: supplying context, calling tools, and deciding when to continue. Different harnesses approach that work differently. You should be able to compare them on your own tasks without rebuilding the surrounding application.

In LiteAgents, the harness is a field in your profile:

import os
from liteagents import LiteAgentClient, LiteAgentOptions, ProfileOptions

profile = ProfileOptions(
    harness="deepagents",  # Try "claude-sdk", "codex", or "pydantic-ai".
    model="my-model",
    model_kwargs={
        "api_base": "https://your-gateway.example/v1",
        "api_key": os.environ["LITELLM_API_KEY"],
    },
)

options = LiteAgentOptions(profile=profile, tools=[lookup_order])
async with LiteAgentClient(options=options) as agent:
    async for message in agent.query("Check the status of order A123."):
        print(message)

This excerpt uses your existing lookup_order tool and the LITELLM_API_KEY environment variable. Set model to the alias configured on your gateway and api_base to its endpoint. LiteAgents sends the alias unchanged; no routing prefix is needed. Once you've installed the harness you want to try, change harness and run the same application.

Install only the harness integrations you plan to use. Extras such as [deepagents] supply that adapter's dependencies and reuse compatible packages in your Python environment. [all] is a convenience for trying every integration; OpenCode also needs its executable. Changing the profile selects which harness runs.

LiteAgents adapts your tools, MCP configuration, and events to each harness. Your application receives the same AssistantMessage, UserMessage, and optional streaming TextDelta types whichever harness you choose. LiteLLM handles model translation internally, so switching harnesses keeps the same model connection. With a gateway, the gateway routes your model alias to its provider. Without a gateway, use a LiteLLM provider/model name and provider credentials; LiteAgents calls that provider through the LiteLLM Python library. Profiles also load from JSON or YAML.

Keep the harness's native controls​

Each harness still runs its own agent loop. Through harness_options, you can use supported native settings such as Deep Agents middleware and backends or Pydantic AI tool timeouts.

Shared configuration makes switching straightforward; native options let you tune the harness you've chosen. Those options remain specific to that harness, and your model must support the settings you request. Switching selects a harness for new runs; it doesn't migrate an ongoing native session.

Add Temporal when you need durable runs​

A simple agent runs locally without Temporal or PostgreSQL. When a job needs to survive a worker restart or keep running after the client disconnects, add Temporal to the profile and run a worker. Your application keeps the same client API.

Durable runs recover recorded operations and checkpoints after worker failure. You can start locally with SQLite, then use self-hosted Temporal or Temporal Cloud. Tools that perform external actions still need idempotency if an interrupted action may be retried.

Try it on your agent​

The SDK is available as a preview. Follow the getting-started guide, or install the package from the latest preview release.

Start with the harness-switching cookbook. It runs the same model, Python tool, MCP tool, and follow-up across all six harness selectors. Add --temporal to try the same task with durability. The cookbook collection also covers streaming, approvals, subagents, and worker recovery.

Try your workflow on another harness and tell us how it goes.

Security​

We shipped 52 security fixes this month, broken down by category:

Major work in security went into authentication:

• Configurable password policies
• Blocking passwords found in known breaches
• Enforced SSO login
• The option to disable logins that use credentials held in environment variables

Master keys. Wiz found 1 in 10 of the LiteLLM instances they scanned running with a blank or default master key. LiteLLM now refuses to start when the master key is unset or still at the default, so a deployment leaning on it will fail to boot on upgrade.

A new disclosure practice, and more CVEs.

• For anything an unauthenticated attacker can reach, we build the fix in a private fork and publish it at the same moment as the release, the backports, and the disclosure.
• We're minting more CVEs than before, mostly low and medium severity.
• The increase comes from a change in publishing policy. Our security posture hasn't changed.
• Subscribe to our GitHub security advisories to get them as they land.

Next month we're at Patch the Planet, an OpenAI and Trail of Bits initiative where security engineers spend a focused week on one project.

Stability​

In September we shipped 583 bug fixes.

The largest share went to the proxy core: scaling, resilience, and the circuit breakers that decide what happens when a dependency fails. If Redis goes down, the gateway should not.

• Coverage. We committed to 80% by the end of September and finished at 83.5%, measured as the share of endpoints and user flows the suite exercises, not lines of code.
• User flows. Up from 402 to 521 covered, with 600 the October target.
• Staffing. One engineer now works only on cost and budget discrepancies, so report them as you hit them. Two more are on open source full time, aiming for fewer than 100 open issues by year end.

Product​

221 feature commits this month.

OCR runs on Rust by default​

Starting with v1.102.0, OCR runs on Rust by default. LITELLM_RUST=0 still falls back to Python. SDK and gateway users don't need to change anything for this migration.

• On a single CPU, median throughput went from 143.6 to 211.7 RPS at 1 MiB, and from 21.6 to 36.2 RPS at 8 MiB.
• These are proxy overhead numbers, measured against a local mock provider with proxy CPU as the bottleneck. Full methodology.
• /messages is next, shipping as opt-in.
• By December 1, we want chat completions, messages, and responses migrated and deployable as a pure Rust Axum server.
• Progress is public at docs.litellm.ai/rust-migration, currently at 5%.

New models​

We added a lot of new models over the past month, across nearly every major provider.

Highlights include OpenAI's GPT-6 family, Claude Opus 5.5 and Claude Fable 5.1, Gemini 3.8 Flash and Grok 4.7, plus Eden AI as a new provider.

Fusion​

litellm/fusion-1 sends your request to a configurable panel of models in parallel, has a judge compare the successful responses, then has that judge synthesize one answer. It works with Chat Completions, Anthropic Messages, and Responses.

It's expensive, and cost and latency both scale with panel size and the models you pick. Keep it for hard problems where quality matters more than cost. See the fusion docs for setup.

liteagents SDK​

liteagents is a beta SDK that lets you swap between agent harnesses without rewriting your agent. Pick the harness in a ProfileOptions block, whether that's the Claude Agent SDK, Deep Agents or Pydantic AI, and query it through one client.

Read more about the liteagents SDK.

Also shipped​

• Self-hosting. A PgBouncer inside the container shares database connections across workers, spend writes can move to a sidecar off the request path, and Helm and Terraform can autoscale on requests and tokens per second per pod rather than only CPU.
• MCP Gateway. The /mcp route now runs semantic search across tools and returns only those matching the query, so we no longer trim long upstream catalogs to fit. Tool permissions are enforced on listing and calling, and session tokens are RS256-signed so an external gateway can verify a token came from LiteLLM.
• Team admin permissions. Budget, TPM, RPM and the approved model list stay with the proxy admin while team admins self-serve keys and models. Rolling out shortly.

We're hiring​

We're hiring across the core gateway; reach us at recruiting@berri.ai, or product@berri.ai for product feedback.

Thank you for using LiteLLM. The LiteLLM Team

LiteLLM supports gemini-3.8-flash-tts and gemini-3.8-flash-lite-tts on day 0 through /v1/audio/speech, on Google AI Studio (gemini/). Both are generally available and replace the 3.1 Flash TTS preview.

Per Google, Flash TTS is built for creative work like games, audiobooks and podcasts, and Flash-Lite TTS is the cheaper tier for high-volume dubbing and voice agents. Both draw on more than 2,000 prebuilt voices, and Flash TTS covers 130 languages with automatic detection.

Launch pricing​

Both models launch at promotional pricing through December 31, 2026, and Google's standard pricing applies from January 1, 2027. LiteLLM tracks cost at the launch rate.

Flash TTS is $0.50 per 1M text input tokens and $9.00 per 1M audio output tokens, and Flash-Lite TTS is $0.50 and $6.00. Batch and Flex run at half these rates and Priority at 1.8x. For comparison, the 3.1 Flash TTS preview was $1.00 in and $20.00 out.

Quick Start​

import litellm

response = litellm.speech(
    model="gemini/gemini-3.8-flash-tts",
    input="Welcome back. Your order shipped this morning.",
    voice="Kore",
)

response.stream_to_file("welcome.wav")

model_list:
  - model_name: gemini-3.8-flash-tts
    litellm_params:
      model: gemini/gemini-3.8-flash-tts
      api_key: os.environ/GEMINI_API_KEY
  - model_name: gemini-3.8-flash-lite-tts
    litellm_params:
      model: gemini/gemini-3.8-flash-lite-tts
      api_key: os.environ/GEMINI_API_KEY

litellm --config /path/to/config.yaml

curl http://0.0.0.0:4000/v1/audio/speech \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini-3.8-flash-tts",
    "input": "Welcome back. Your order shipped this morning.",
    "voice": "Kore"
  }' \
  --output welcome.wav

Notes​

Input is capped at 8,192 tokens per request and output at 16,384, so split long scripts such as audiobook chapters across calls. Pass a prebuilt voice by name in voice; see Google's speech generation guide for the list.

Feedback​

Running Gemini 3.8 TTS through LiteLLM and hitting something unexpected? Share it on GitHub discussion #42776.

An engineer asks for an API key for a new project. You need to choose its models, set a budget, and assign it to a team. As usage grows, you need to check spending and adjust those limits.

LiteAdmin MCP lets your agent handle these tasks through your gateway's management API. Connect it to an MCP client or a custom agent. We built LiteAdmin, our Slack admin agent, on the same connector.

Connect your agent to your gateway​

Connect LiteAdmin MCP to a client such as Claude Code or Cursor, or to your own agent. Your client provides the conversation and model; the connector calls your gateway's management API with your admin credential.

Through that connection, you can:

• Create and manage virtual keys: Set model access and spending limits.
• Add model deployments: Use credentials configured on your gateway.
• Manage teams and budgets: Update team membership and budgets.
• Inspect usage: Check spending and request logs.

The connector includes 65 reviewed admin operations and exposes the ones your gateway supports.

Set up access for a project​

To set up a project for Engineering, ask:

Create a key for Engineering with a $100 monthly budget.

Include the models the project needs and an alias to identify the key. The agent can look up Engineering and create the key with the limits you specify.

Once the project is running, ask for its recorded spend. You can update the key's budget or the team's budget; specify which one in your request.

To add a model deployment, provide its public name, provider/model ID, and credential reference:

Add a model named support-chat using openai/gpt-4.1 and the existing gateway credential openai-production.

Use a stored credential name and keep provider API keys out of chat. Your gateway needs provider access and a database configured to store models. Follow the model setup requirements, then test inference after adding the deployment.

Choose the tools your agent can use​

Connect with your own LiteLLM proxy-admin credential. Your gateway permissions apply to management requests, and you can restrict the connector to specific tools or enable read-only mode.

Start with team and spend lookups, then enable key creation when you're ready to make changes.

LiteAdmin: a Slack agent built on Admin MCP​

LiteAdmin (the LiteLLM Admin Agent) uses the same MCP server to manage your gateway from Slack. The app provides the conversation and sign-in; Admin MCP provides the gateway tools.

Host the agent, connect your gateway, and choose the model it uses. The app includes the MCP connector, and each installation connects one gateway to one Slack workspace.

Open LiteLLM Admin in Slack, send connect, and follow the private link to sign in with your own admin account. Your deployment can use SSO or a personal admin key entered on a browser page. Then ask:

Show Engineering's current spend and budget.

After checking the spend, request a change:

Increase Engineering's monthly budget to $500.

You can follow up about the team's keys or usage in the same DM, from your laptop or phone.

Get started​

Both projects are open source. Choose where you want to work:

• LiteAdmin MCP: Follow the MCP setup guide to connect Claude, Codex, or another MCP client.
• LiteAdmin for Slack: Follow the Slack app setup guide to deploy with Docker Compose or Render.

Once connected, try “List my teams and their current budgets.”

LiteLLM now supports GPT-6 Sol and GPT-6 Luna. Route traffic to them through the LiteLLM AI Gateway with the same config you use for every other OpenAI model.

Sol and Luna join GPT-6 Astra, trained with the same methods and priced for work at scale. gpt-6-sol is built for complex coding and agentic workflows at $2 input and $10 output per 1M tokens, and gpt-6-luna is the high-volume tier at $0.10 and $0.50. OpenAI prices both at half their GPT-5.6 counterparts' current rates. Per OpenAI, Sol at xhigh scores 33.2% on AutomationBench at $0.27 a task and 68.8% on DeepSWE v1.1 at max, while Luna reaches 66.6% on DeepSWE. Both keep a 1,050,000-token context window with 922K input and 128K output, take text and image input, and run reasoning effort from none to max, defaulting to medium. There is no GPT-6 Terra; Astra stays the top of the range.

Usage​

model_list:
  - model_name: gpt-6-sol
    litellm_params:
      model: openai/gpt-6-sol
      api_key: os.environ/OPENAI_API_KEY
  - model_name: gpt-6-luna
    litellm_params:
      model: openai/gpt-6-luna
      api_key: os.environ/OPENAI_API_KEY

docker run -d \
  -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:main-latest \
  --config /app/config.yaml

curl -X POST "http://0.0.0.0:4000/chat/completions" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-6-sol",
    "messages": [
      {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
    "reasoning_effort": "high"
  }'

from litellm import completion

response = completion(
    model="openai/gpt-6-sol",
    messages=[
        {"role": "user", "content": "Write a Python function to check if a number is prime."}
    ],
    reasoning_effort="high",
)

print(response.choices[0].message.content)

# gpt-6-luna for focused, high-volume work
response = completion(
    model="openai/gpt-6-luna",
    messages=[
        {"role": "user", "content": "Classify this ticket as bug, feature, or question."}
    ],
    reasoning_effort="none",
    temperature=0,
)

print(response.choices[0].message.content)

Responses API​

For agentic and multi-turn workflows, use /v1/responses to preserve reasoning state across turns.

curl -X POST "http://0.0.0.0:4000/v1/responses" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -d '{
    "model": "gpt-6-sol",
    "input": "Plan and write a Python script that scrapes a webpage and summarizes it."
  }'

Pricing​

Prices are per 1M tokens (USD), shown as short context (≤272K tokens) / long context (>272K tokens).

Flex and Batch run at half these rates and Priority at double; LiteLLM tracks all of them from the same cost map row.

Notes​

temperature only applies when reasoning is off, so send reasoning_effort="none" alongside it, as in the Luna example above. Without it LiteLLM drops or refuses the parameter, since the default effort is medium.

Both models accept OpenAI's explicit prompt cache breakpoints, and LiteLLM passes them through on /chat/completions, /responses and /v1/messages; see prompt caching. OpenAI also says changing reasoning effort or tools mid-conversation no longer breaks the cache, which matters if you route different effort levels to the same deployment.

Feedback​

Running GPT-6 Sol or Luna through LiteLLM and hitting something unexpected? Share it on GitHub discussion #42523.

LiteLLM now supports Claude Opus 5.5 on Day 0. Use it across Anthropic, Bedrock, Gemini Enterprise Agent Platform, and Azure through the LiteLLM AI Gateway. Call it with the same OpenAI-compatible request you already use, and track spend, rate limits, and logging in one place.

What's new in Opus 5.5​

Key changes (details from Anthropic):

• 20% cheaper than Opus 5: $4 / MTok input and $20 / MTok output, down from $5 and $25
• Cheaper cache reads: $0.20 / MTok, with 5-minute writes at $5 and 1-hour writes at $8
• Stronger agentic coding: Terminal-Bench 4.0 at 66.4% and OSWorld 2.0 at 81.8%
• Faster and leaner: Anthropic reports output more than 30% faster, using fewer tokens per task
• Thinking is always on: effort is the only control, and its default is medium
• Fast mode: about 2x the base price, down from Opus 5's $10 / $50

Context window and max output are unchanged from Opus 5 at 1M and 128K.

Enabling Opus 5.5​

Pricing landed in PR #42489, and most proxies do not need to upgrade at all. On the default remote cost map, open the Price Data tab under Models + Endpoints in the UI and click Reload Price Data (or POST /reload/model_cost_map as a proxy admin), on any version v1.76.0 or newer.

The exception is LITELLM_LOCAL_MODEL_COST_MAP=true, which bakes the cost map into the image and puts it out of the Reload button's reach. That path needs an image built after #42489 merged.

Usage​

Pick your provider below. Each tab wires up claude-opus-5-5 for that provider; the request you send afterward is identical everywhere.

model_list:
  - model_name: claude-opus-5-5
    litellm_params:
      model: anthropic/claude-opus-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

docker run -d \
  -p 4000:4000 \
  -e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.103.0-rc.1 \
  --config /app/config.yaml

model_list:
  - model_name: claude-opus-5-5
    litellm_params:
      model: bedrock/anthropic.claude-opus-5-5
      aws_access_key_id: os.environ/AWS_ACCESS_KEY_ID
      aws_secret_access_key: os.environ/AWS_SECRET_ACCESS_KEY
      aws_region_name: us-east-1

docker run -d \
  -p 4000:4000 \
  -e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
  -e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.103.0-rc.1 \
  --config /app/config.yaml

model_list:
  - model_name: claude-opus-5-5
    litellm_params:
      model: vertex_ai/claude-opus-5-5
      vertex_project: os.environ/VERTEX_PROJECT
      vertex_location: global

docker run -d \
  -p 4000:4000 \
  -e VERTEX_PROJECT=$VERTEX_PROJECT \
  -e GOOGLE_APPLICATION_CREDENTIALS=/app/credentials.json \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -v $(pwd)/credentials.json:/app/credentials.json \
  ghcr.io/berriai/litellm:v1.103.0-rc.1 \
  --config /app/config.yaml

model_list:
  - model_name: claude-opus-5-5
    litellm_params:
      model: azure_ai/claude-opus-5-5
      api_key: os.environ/AZURE_AI_API_KEY
      api_base: os.environ/AZURE_AI_API_BASE  # https://<resource>.services.ai.azure.com

docker run -d \
  -p 4000:4000 \
  -e AZURE_AI_API_KEY=$AZURE_AI_API_KEY \
  -e AZURE_AI_API_BASE=$AZURE_AI_API_BASE \
  -v $(pwd)/config.yaml:/app/config.yaml \
  ghcr.io/berriai/litellm:v1.103.0-rc.1 \
  --config /app/config.yaml

3. Test it!

The request is the same regardless of which provider you configured above:

curl --location 'http://0.0.0.0:4000/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-opus-5-5",
  "messages": [
    {
      "role": "user",
      "content": "what llm are you"
    }
  ]
}'

Fast Mode​

Opus 5.5 runs faster with speed: "fast", billed at $8 / MTok input and $40 / MTok output, 2x the standard rate and down from the $10 / $50 on Opus 5. LiteLLM adds the fast-mode-2026-02-01 beta header and tracks the premium in cost calculations automatically.

curl --location 'http://0.0.0.0:4000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $LITELLM_KEY' \
--data '{
  "model": "claude-opus-5-5",
  "messages": [
    {
      "role": "user",
      "content": "Refactor this module..."
    }
  ],
  "max_tokens": 4096,
  "speed": "fast"
}'

Worth knowing​

Forced tool use returns a 400. With drop_params on, LiteLLM downgrades tool_choice to auto and logs a warning, so the call succeeds and the model is free not to call your tool.

Thinking cannot be disabled, and the default effort is now medium where Opus 5's was high, so a request that never set effort thinks less than it used to.

Thinking blocks are tied to the model. Only Fable 5.1 and Mythos 5.1 read Opus 5.5's, so a fallback to Sonnet or Haiku runs those turns without the reasoning.

Feedback​

Running Claude Opus 5.5 through LiteLLM and hitting something unexpected? Share it on GitHub discussion #42494.

What Fireworks AI gives you​

Fireworks AI serves open models on fast inference infrastructure. Families like DeepSeek, Qwen, GLM, Kimi, MiniMax, and GPT-OSS are available as serverless endpoints you can call immediately with an API key. As your needs grow you can host a model inside your own Fireworks account, reserve a dedicated deployment for consistent throughput, or point at a Fireworks router that keeps you on the current version of a model family. Open models run fast on Fireworks and cost a fraction of frontier pricing, and because the weights are open you can fine-tune one on your own data and serve the result from the same platform.

What LiteLLM gives you​

LiteLLM is an open source AI gateway. You run it in front of your models, and it gives every provider one OpenAI-compatible API. Your application sends a normal OpenAI request to one endpoint, and the gateway handles the provider-specific details. It also tracks token usage and cost for every request, issues keys with budgets attached, and can fall back to another model when one is busy.

What you get together​

Your application code speaks OpenAI format to a single endpoint and never learns that Fireworks is behind it. Changing which model serves a route becomes a line in a config file, so you can move traffic to a cheaper or faster model without a deploy. Spend shows up per key and per team from the first request. When Fireworks adds a model you want, or you move a route onto a version fine-tuned on your own data, you pick it up by editing that one line.

What you need​

A Fireworks API key from the Fireworks dashboard, Python 3.10 or newer, and a couple of minutes. Install the gateway with pip install 'litellm[proxy]'.

1. Write a config file​

Create config.yaml with one model. The model_name is the name your application will use. The model underneath it is the real Fireworks model, and the fireworks_ai/ prefix tells the gateway which provider to route to.

model_list:
  - model_name: my-model
    litellm_params:
      model: fireworks_ai/deepseek-v4p1-flash
      api_key: os.environ/FIREWORKS_AI_API_KEY

Writing os.environ/FIREWORKS_AI_API_KEY keeps the key out of the file. The gateway reads it from the environment at startup.

2. Start the gateway​

export FIREWORKS_AI_API_KEY="your-key-here"
litellm --config config.yaml

The gateway comes up on http://0.0.0.0:4000.

3. Send a request​

Any OpenAI-compatible client works. With curl:

curl http://0.0.0.0:4000/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "my-model",
    "messages": [{"role": "user", "content": "what llm are you"}]
  }'

Or with the OpenAI Python SDK, changing only the base_url:

import openai

client = openai.OpenAI(
    api_key="anything",
    base_url="http://0.0.0.0:4000",
)

response = client.chat.completions.create(
    model="my-model",
    messages=[{"role": "user", "content": "write a haiku about gateways"}],
)

print(response.choices[0].message.content)

Note the model name in the request is my-model, the alias you chose, and added earlier in the config.yaml file.

4. Add a second model and a fallback​

You can add as many models as you like. In this tutorial they live in config.yaml, so add another entry to model_list, then tell the router to try the second model when the first is unavailable. Once you connect a database you can also add and edit models from the gateway dashboard without restarting.

model_list:
  - model_name: my-model
    litellm_params:
      model: fireworks_ai/deepseek-v4p1-flash
      api_key: os.environ/FIREWORKS_AI_API_KEY

  - model_name: my-backup
    litellm_params:
      model: fireworks_ai/glm-5p3-flash
      api_key: os.environ/FIREWORKS_AI_API_KEY

router_settings:
  fallbacks: [{"my-model": ["my-backup"]}]

Restart the gateway and your application keeps calling my-model. Rate limits and transient errors now route to the backup automatically.

Where to go next​

Once the basics are running, the same config file is where you reach the rest of the platform. Account-hosted models use their full accounts/fireworks/models/... path, dedicated deployments add an api_base, and Fireworks routers take a routers/ prefix. Requests to /v1/responses reach the native Fireworks Responses API, so server-side MCP tools and response chaining work through the gateway. Document inlining lets models without vision support read documents and images. Embeddings, reranking, and audio transcription route through the same key and the same spend tracking.

To hand out scoped keys with their own budgets, connect a Postgres database and set a master key; see virtual keys. The full provider reference lives in the Fireworks AI docs.

LiteLLM supports mimo-v2.6-pro and mimo-v2.6-flash on day 0, and prices both on Xiaomi's own endpoint for the first time.

Pricing​

Per 1M tokens, Pro is $0.435 input and $0.87 output, Flash is $0.14 and $0.28. V2.6 costs what V2.5 cost; Xiaomi kept the rates and raised the model.

Pro reads cached input at $0.0036, about a 121st of its input rate, where most providers in the same cost map charge a tenth. Cache writes are free for now.

Thinking is on by default​

Every V2.6 model reasons unless told otherwise, and Xiaomi controls that with thinking.type rather than reasoning_effort. LiteLLM treats xiaomi_mimo as an OpenAI-compatible provider and does not map thinking for it yet, so pass it through explicitly with allowed_openai_params=["thinking"].

In multi-turn tool calling, the reasoning_content from a prior assistant turn has to go back in the next request or the API returns a 400.

Usage​

from litellm import completion

response = completion(
    model="xiaomi_mimo/mimo-v2.6-pro",
    messages=[{"role": "user", "content": "Refactor this migration script."}],
    thinking={"type": "disabled"},          # on by default
    allowed_openai_params=["thinking"],
)

print(response.choices[0].message.content)

model_list:
  - model_name: mimo-v2.6-pro
    litellm_params:
      model: xiaomi_mimo/mimo-v2.6-pro
      api_key: os.environ/XIAOMI_MIMO_API_KEY
      allowed_openai_params: ["thinking"]

  - model_name: mimo-v2.6-flash
    litellm_params:
      model: xiaomi_mimo/mimo-v2.6-flash
      api_key: os.environ/XIAOMI_MIMO_API_KEY

Hit Reload Model Cost Map in the Admin UI, or POST /reload/model_cost_map, to pick the rows up without a redeploy on v1.76.0 and above.

If you are on V2.5​

Xiaomi deprecates mimo-v2.5-pro and mimo-v2.5 at 10:00 Beijing time on October 21, 2026.

Feedback​

Running MiMo V2.6 through LiteLLM and hitting something unexpected? Share it on GitHub discussions.

Anthropic is moving Claude Code auto mode's safety classifier from the client to the Claude API. Starting with Claude Code v2.1.278, released September 19, sessions on Enterprise plans and Claude API accounts ask the server to run those checks as part of their own model requests, and Anthropic does not charge for the checks when the server performs them. Anthropic told us the rollout started on September 18 and is gradual, beginning with the Claude Code CLI and VS Code extension and followed by the desktop app and Claude Code on the web over the following week, and that on September 25 auto mode becomes the default permission mode in Claude Code. Today the built-in default is auto on Pro, Max and Team plans and Manual on Enterprise plans and Claude API keys, the accounts that typically sit behind a gateway, per Anthropic's permission modes reference.

Server-side auto mode depends on a contract between Claude Code and the API that some gateways did not preserve, LiteLLM included. The fix first shipped in the dev release cut on Tuesday, September 22, and is now in the v1.104.0 stable release and in patch releases of earlier lines. This post explains what Claude Code needs from an AI Gateway, what LiteLLM was doing wrong, what changed, which release carries it, and how to confirm your deployment is ready.

What server-side auto mode needs from an AI Gateway​

Claude Code sends a safeguards field in the /v1/messages request body, an array with one dangerous_tool_use entry that carries the session's permission mode, and it sets dangerous-tool-use-2026-09-03 in the anthropic-beta header. The API answers with a safeguard_results field: one dangerous_tool_use entry whose status.type is available and whose status.tool_uses is keyed by tool use ID, each marked evaluated with an outcome such as not_flagged. When the response is streamed, safeguard_results arrives inside the delta of the final message_delta event.

For server-side auto mode to run, a gateway has to forward request headers and body fields as they are, including ones it does not recognize such as safeguards, and return responses and streaming events without dropping keys such as safeguard_results or rewriting tool use IDs. Anthropic's gateway compatibility guide spells this out.

If any of that is dropped or rewritten, the server's checks never reach the session, and Claude Code keeps using its own classifier requests, billed as before. Before the first action it would check that way, Claude Code holds the action and shows this notice, naming the gateway:

We're changing auto mode to no longer charge for classifier requests in Claude Code. However, this session isn't eligible because your requests go through <your gateway>, which isn't compatible with this update. Nothing breaks: auto mode keeps working, and its classifier requests are billed as before. To fix it and access the new version of auto mode, ask your gateway to implement: https://code.claude.com/docs/en/auto-mode-classifier-billing

Nothing breaks when this happens. Users keep the auto mode they have today and keep paying for the classifier calls. Anthropic has told us that new Claude Code releases keep the client-side classifier until at least October 23, 2026, that releases after that date only support server-side auto mode, and that from then on auto mode is not available behind a gateway that does not support the server-side classifier, so gateways have a window to catch up. Anthropic's notice reference covers who sees the notice and what it means.

What LiteLLM was doing wrong​

LiteLLM's native /v1/messages endpoint builds the outbound request from an allowlist of known Anthropic Messages parameters so that the same endpoint can front Claude on Bedrock, Vertex AI, Azure AI and non-Anthropic models. safeguards was not on that list, so it was silently dropped before the request left the proxy.

Anthropic therefore received a request with no safeguards, returned no evaluated safeguard_results, and Claude Code fell back to the paid client-side classifier. Running Claude Code v2.1.278 in auto mode against a LiteLLM release without the fix shows the notice above with your proxy's address in it, and /status reports the Auto mode server row as Disabled.

The raw pass-through route, POST /anthropic/v1/messages, was never affected. It forwards the body and headers verbatim, and it already carried safeguards and the full beta header before the fix. We confirmed this by running Anthropic's gateway check against a build from before the fix: the pass-through route passes and the native route fails.

What changed​

PR #42152, merged into main on September 21, 2026, makes the native /v1/messages route preserve the contract. safeguards is now a recognized request parameter and is forwarded as sent. When the resolved provider is first-party anthropic, the anthropic-beta header is forwarded unchanged instead of being filtered against the known-betas list; requests to Claude on Bedrock, Vertex AI and Azure AI keep the existing filtering because those providers still reject unknown flags. safeguard_results is declared on the response and streaming chunk types, and it is returned unchanged in both the JSON response and the final message_delta event when streaming. LiteLLM does not rewrite tool use IDs on this route, so safeguard_results entries still match the tool uses they refer to.

When /v1/messages is used to reach a non-Anthropic model through the adapter path, safeguards is stripped before the request is translated so those backends do not return a 400.

Non-Claude models behind the proxy can be covered too from v1.107.0. With safeguards_classifier_model set in general_settings, the proxy generates the auto mode verdicts itself for every non-Claude model behind /v1/messages, such as an open model on Fireworks, a Gemini deployment, an openai/ deployment, or DeepSeek on its own Anthropic-compatible endpoint, and for bedrock/converse/ models, Claude included. It asks the configured classifier deployment to judge each tool call of the response and returns safeguard_results the way the Anthropic API does, so auto mode stays on server review when Claude Code is pointed at an open model. Setup, cost and the failure behavior are in Claude Code auto mode on non-Claude models.

This fix covers LiteLLM's route to the Anthropic API. Claude Code also asks for server-side checks on Amazon Bedrock and Google Cloud's Vertex AI, and PR #42288, merged on September 21, 2026, covers those too. On /v1/messages, Claude on Bedrock InvokeModel (for example bedrock/us.anthropic.claude-sonnet-5) and Claude on Vertex AI now get safeguards and the dangerous-tool-use-2026-09-03 beta forwarded, and safeguard_results comes back unchanged. That change is in v1.99.3, v1.100.2, v1.101.1, v1.102.1, v1.103.2 and v1.104.0; v1.101.0 and v1.103.0 do not have it. Claude on Bedrock Mantle (bedrock_mantle/anthropic.claude-sonnet-5) is served through Mantle's native Anthropic Messages API from v1.104.0 and preserves the contract there too.

Bedrock itself does not answer on the Converse route, bedrock/converse/<model>. LiteLLM translates it through the chat completions adapter, which drops safeguards, and Bedrock's ConverseStream API does not return safeguard_results even when they are sent, so Claude Code keeps its paid classifier there unless safeguards_classifier_model is set, in which case the proxy answers for Converse models, Claude included. On releases without the fix the verification request below fails on this route with a 400, safeguards: Extra inputs are not permitted. Without the setting, point Claude Code at a bedrock/<inference profile> deployment, or a bedrock_mantle/ deployment on v1.104.0 or later, instead. Microsoft Foundry is tracked separately and not covered by this post.

It first shipped in the dev release cut from main on Tuesday, September 22, ahead of Anthropic's September 25 default change. Dev releases are pre-release builds published to PyPI, Docker Hub and GitHub releases as -dev.N tags, so this one is v1.104.0-dev.1 by the current numbering (litellm==1.104.0.dev1 on PyPI). The fix for the Anthropic API route is now in the v1.104.0 stable release and was backported to v1.99.3, v1.100.2, v1.101.1, v1.102.1 and v1.103.2. Claude Code sessions routed through the native /v1/messages endpoint on any earlier LiteLLM release see the notice and keep using the client-side classifier until you upgrade. If you would rather your users not see the notice in the meantime, Anthropic documents setting CLAUDE_CODE_AUTO_MODE_SERVER=0 in the environment Claude Code starts from, which tells it not to ask the gateway for server-side checks. That only hides the notice: Claude Code keeps making the same paid classifier requests until you upgrade.

How to verify your deployment​

Send a request that mirrors what Claude Code sends, with a forced tool call so the server has something to evaluate, and check the response for safeguard_results. The model has to be a deployment that LiteLLM routes to the Anthropic API, Bedrock InvokeModel, Bedrock Mantle or Vertex AI; replace claude-sonnet-5 with that deployment's model name on your proxy.

curl -s "$LITELLM_PROXY_URL/v1/messages" \
  -H "Authorization: Bearer $LITELLM_API_KEY" \
  -H "content-type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -H "anthropic-beta: dangerous-tool-use-2026-09-03" \
  -d '{
    "model": "claude-sonnet-5",
    "max_tokens": 256,
    "safeguards": [{"type": "dangerous_tool_use", "classifier_context": {"v": 1, "permission_mode": "auto"}}],
    "tools": [{"name": "Bash", "description": "Runs a shell command", "input_schema": {"type": "object", "properties": {"command": {"type": "string"}}, "required": ["command"]}}],
    "tool_choice": {"type": "tool", "name": "Bash"},
    "messages": [{"role": "user", "content": "Run: echo hello"}]
  }' | jq '{safeguard_results, tool_use_ids: [.content[] | select(.type == "tool_use") | .id]}'

On a release with the fix, the tool use ID in safeguard_results matches the one in the response content:

{
  "safeguard_results": [
    {
      "type": "dangerous_tool_use",
      "status": {
        "type": "available",
        "tool_uses": {
          "toolu_01V9Z5KXn3SU71Fzr5cquHLi": {
            "type": "evaluated",
            "outcome": "not_flagged"
          }
        }
      }
    }
  ],
  "tool_use_ids": [
    "toolu_01V9Z5KXn3SU71Fzr5cquHLi"
  ]
}

On a proxy without the fix, safeguard_results comes back empty ([]) or missing, because safeguards never reached the provider.

Anthropic shared a gateway check script with us that sends the same request non-streaming and streaming and checks that every tool use ID comes back evaluated. Both legs pass against a release with the fix. You can also check from Claude Code itself: start a session through your proxy in auto mode, send a prompt and wait for the reply, then run /status and look for the Auto mode server row reading Enabled. Before the first model response the row reads Enabled on any proxy, so check it after a reply. In non-interactive mode with -p --output-format stream-json, the notice above arrives as a system message at warning level, so a scripted check can grep for it.

If you use the /anthropic/v1/messages pass-through route today, no action is needed.

Frequently Asked Questions​

Does this change how LiteLLM handles beta headers for Bedrock, Vertex AI or Azure AI?​

Beta header filtering still applies when the resolved provider is anything other than first-party anthropic, because those providers reject unknown beta flags, so the allowlist in anthropic_beta_headers_config.json remains the source of truth for them. dangerous-tool-use-2026-09-03 is on that allowlist for Bedrock InvokeModel and Vertex AI in the releases listed above, and for Bedrock Mantle from v1.104.0, which is how server-side auto mode runs on those routes. Bedrock Converse maps it to nothing, so it is not forwarded there. Only requests bound for the Anthropic API forward the whole header unchanged.

Will my Claude Code users be broken before I upgrade?​

No. Claude Code detects that the server's checks are not reaching the session and keeps using its own classifier. Users see the notice, keep the current experience, and keep being billed for classifier calls until you upgrade. Anthropic has told us new Claude Code releases keep the client-side classifier until at least October 23, 2026, and that releases after that date need the server-side classifier for auto mode, so upgrade before then.

Is this available in LiteLLM OSS?​

Yes. The fix is in LiteLLM OSS (Apache 2.0) and requires no configuration. LiteLLM Enterprise adds SSO/SCIM, air-gapped deployment, 24/7 SLA support and advanced guardrails on top.

Conclusion​

An AI Gateway in front of Claude Code has to forward provider contracts it did not exist for when they were designed. The safeguards field is one of those, and LiteLLM's native /v1/messages route now passes it through unchanged on the way to the Anthropic API, Bedrock InvokeModel, Bedrock Mantle and Vertex AI. Upgrade to a release with the fix, keep Claude Code off bedrock/converse/ deployments, run the check above, and your users get server-side auto mode at no cost.

Recommended Reading​

• Claude Code with LiteLLM AI Gateway
• Claude Code auto mode on non-Claude models
• Claude Code: managing Anthropic beta headers
• Anthropic pass-through endpoints
• Anthropic: auto mode classifier request charges
• Anthropic: LLM gateway compatibility guide

LiteLLM supports grok-4.7 on day 0, on /chat/completions and /responses. It keeps Grok 4.6's prices, limits and capabilities.

SpaceXAI, the company formerly called xAI, built 4.7 and reports the model is better at checking its own work and holding long context. Against Grok 4.6, Terminal-Bench 4.0 goes from 20.3% to 38.0% and EEBench from 53.0% to 64.0%.

Pricing​

Per 1M tokens: $2.00 input, $0.50 cached, $6.00 output, identical to Grok 4.6. Every rate doubles past 200K input tokens, to $4.00, $1.00 and $12.00, so a long-context request costs twice what the headline suggests.

Usage​

from litellm import completion

response = completion(
    model="xai/grok-4.7",
    messages=[{"role": "user", "content": "Refactor this migration script."}],
    reasoning_effort="xhigh",  # low | medium | high (default) | xhigh
)

print(response.choices[0].message.content)

model_list:
  - model_name: grok-4.7
    litellm_params:
      model: xai/grok-4.7
      api_key: os.environ/XAI_API_KEY

  - model_name: grok-4.7-openrouter
    litellm_params:
      model: openrouter/x-ai/grok-4.7
      api_key: os.environ/OPENROUTER_API_KEY

Pricing landed in PR #42264. Hit Reload Model Cost Map in the Admin UI, or POST /reload/model_cost_map, to pick it up without a redeploy on v1.76.0 and above.

Feedback​

Running Grok 4.7 through LiteLLM and hitting something unexpected? Share it on GitHub discussion #42287.

An Auto Router pays for classification before the selected model can answer. In our benchmark, TypeSafe JEV classified requests 5.43x as fast as Haiku, comparing median classifier latency: 126.81 ms versus 688.40 ms. Registry-priced classifier cost was 96.12% lower, rounded to 96% in the title

JEV matched our benchmark's expected tiers on 95.00% of calls, versus 73.75% for Haiku. That result depends on the prompts, tier definitions, instructions and context used here. The expected tiers were authored with the synthetic prompts, without independent review. This comparison does not establish general classification accuracy or the quality of the final answers

The AI Gateway has to pay for the routing decision​

Classifying a greeting and a complex debugging request lets you send each to a different model. A general-purpose LLM can make that decision, but its network round trip, input tokens and structured output add latency and cost before the completion begins

We compared the existing LLM classifier path with classifier_type: jev, using the same authored tier rubric and opening instructions. JEV uses TypeSafe System One Choice evaluation. It returns a tier choice with confidence and probabilities, and the existing Auto Router selects a model from that tier. Tier pools, fallback and the completion path stay in the same router

The question we measured was narrow: how quickly and cheaply can each classifier reproduce the labels we assigned to these prompts? We did not grade generated answers or establish general LLM-quality equivalence

What we measured​

The benchmark ran on September 18, 2026 UTC, with jev-latest first resolved to jev-1.13.0 and then pinned. The comparison model was anthropic/claude-haiku-4-5-20251001. Each classified the same 80 cases three times, giving 240 measured calls per classifier

All measured attempts succeeded, so all-attempt and successful-call latency statistics are identical. Calls with an incorrect tier remain in both latency and cost totals

How we calculate 5.43x as fast​

The title compares the median of all 240 JEV calls with the median of all 240 Haiku calls. Using unrounded measurements:

p50 speed ratio = Haiku_p50 / JEV_p50
                = 688.395634 ms / 126.814470 ms
                = 5.43x

This is a ratio of aggregate statistics, not an average of per-request ratios. The same calculation gives 3.88x as fast at p95 (231.16 ms versus 896.94 ms) and 5.21x as fast by mean latency (138.38 ms versus 720.71 ms). These ratios measure classification time, not end-to-end completion speed or throughput under load

What matching the expected tiers tells us​

The match rate counts exact matches with a frozen, authored tier label. JEV's 95% interval was 90.00% to 98.75%, and Haiku's was 64.16% to 82.92%. Labels and prompts came from the same author, without independent annotation or blind adjudication

Tier selection depends on both the user's request and the classifier's instructions. A different request mix, rubric, tier boundary or conversation history can change the result for either classifier. We tested one fixed configuration, without independently tuning either prompt or measuring sensitivity to alternate instructions. Matching these labels also does not prove that the selected completion model can answer a request well

The classifiers agreed with each other on 189/240 calls, or 78.75%, with an interval of 69.58% to 87.08%. Agreement ignores whether either classifier matched the authored label

An empirical interval of 100% to 100% means every case in that sampled subset matched. It does not establish perfect accuracy on unseen prompts. There are 80 independent case clusters, with correlated repeats

JEV consistently assigned four cases to a lower tier than the authored label: M07, M13 and M15 went from MEDIUM to SIMPLE, and C10 went from COMPLEX to MEDIUM. Haiku's mismatches also assigned lower tiers: 37 MEDIUM observations became SIMPLE, 12 COMPLEX became MEDIUM, and 14 REASONING became COMPLEX. The largest difference is on the subjective MEDIUM boundary

The boundary subset's match-rate difference was 12.50 points with a 0 to 37.50 point interval, which includes zero. JEV matched fewer expected tiers on long cases than on short cases. Both findings matter when choosing prompts for your own evaluation

Per-tier latency​

Full per-tier and per-repeat latency, cost, confidence intervals, confusion matrices and mismatches are in the downloadable metrics.json

Cost savings stop at the classifier boundary​

The cost comparison uses observed tokens multiplied by the archived LiteLLM registry prices. It is classifier cost, excluding the completion model, embeddings, warmups and evaluation judges

The resulting saving is 100 * (1 - 0.007706664 / 0.198534) = 96.118%. Every measured response was independently repriced against the archived registry and matched the classifier's reported cost. Prices are the registry values captured for this run. Provider invoices, taxes, discounts and plan entitlements were not checked

The corrected run's excluded resolution and warmup calls cost $0.002478554 at those prices. An earlier setup resolution call is retained separately. They are overhead outside the 240-call comparison

Configure JEV in the existing Auto Router​

Provision TYPESAFE_API_KEY in the proxy's server environment. TYPESAFE_API_BASE is optional and defaults to https://api.typesafe.ai. Keep provider credentials on the server, and let clients call the router with a LiteLLM virtual key

Add this entry beside the existing tier deployments in model_list, replacing the tier values with your deployed model names:

- model_name: jev-router
  litellm_params:
    model: auto_router/complexity_router
    complexity_router_default_model: {{openai_large}}
    complexity_router_config:
      tiers:
        SIMPLE: {{openai_small}}
        MEDIUM: {{openai_large}}
        COMPLEX: {{anthropic}}
        REASONING: {{anthropic_large}}
      classifier_type: jev
      jev_classifier_config:
        model: jev-latest
        timeout_ms: 3000
        circuit_breaker_enabled: true
        circuit_breaker_cooldown_seconds: 30
      classifier_fallback: default_model
      classifier_context_window_size: 3
      classifier_context_budget_chars: 8000
      classifier_context_include_assistant_turns: false

This deployment example uses built-in tier criteria, a three-second deadline and a circuit breaker. It explicitly chooses default-model fallback. The benchmark used a shared custom rubric, a ten-second deadline, assistant context and disabled breakers to compare every measured attempt

In the dashboard, create or edit an Auto Router under Models + Endpoints, then select JEV Classifier under Classification Method. The form exposes model, timeout, circuit breaker, context and fallback settings. Custom JEV Instructions replace the built-in instructions and follow the Enterprise custom-classifier policy. The setup guide covers the full create, test and edit flow

JEV receives the current ask and classifier context in a System One state, with one questions.tier Choice question. Built-in tiers use shipped criteria. With tier_definitions, the custom descriptions become the Choice criteria. The classifier context budget bounds prior-turn text, while the current ask and extracted system text sit outside it

Like the LLM classifier, JEV sends up to three prior user turns by default, within an 8,000-character budget. That history goes to the configured TypeSafe endpoint. Set classifier_context_window_size: 0 to omit it, including when upgrading an existing JEV router

Failure handling in a production-grade AI Gateway​

A reliable AI Gateway needs a defined route when its classifier cannot answer. JEV uses the existing fallback on a timeout, HTTP failure, malformed response, unknown tier or open circuit. Built-in tiers default to heuristic fallback. Setting classifier_fallback: default_model uses the configured default, while custom tiers can choose a fallback_tier

To keep routing resilient to classifier timeouts, the enabled circuit breaker skips JEV calls during the 30-second default cooldown after a recognized timeout. One request then probes recovery. Success closes it, and failure starts another cooldown. The state is local to a router instance in one process, so it does not coordinate provider load across workers. Non-timeout errors while closed fall back without opening the circuit

The supplied live gateway evidence used a real local proxy, PostgreSQL and provider calls. A normal completion and Test Routing succeeded. A separate 1 ms deadline configuration produced a successful completion with default_model_fallback recorded. The logs did not retain the exception subtype, and no JEV usage returned for that cancellation, so receipt or billing by the provider remains unverified

All downstream tier aliases in this gateway check used the same Haiku model. That demonstrates tier selection and execution, without measuring quality or savings from different completion models. Streaming, tool-call completions, custom tier names, all authorization roles and multi-worker accounting are outside this proof

Test Routing can spend money​

Test Routing stops after classification and does not send a completion to the selected model. A JEV classification can still incur a charge. Test Connection additionally probes model dependencies and checks that the JEV result actually came from JEV rather than fallback

In the supplied gateway check, Test Routing persisted one classifier row costing $0.000018186. The normal completion persisted a $0.000018060 classifier row and a $0.000064000 downstream row. The completion's routing metadata carried the classifier cost for attribution, but its spend stayed separate

For accounting, sum the stored row spends once. Adding routing_decision.classifier_cost again would double count the classifier. Successful JEV decisions include the resolved classifier model, probabilities, confidence and registry-priced cost where available. Missing usage or pricing leaves cost unknown. A usable HTTP response can be logged even when its choice later fails validation, so inspect classifier spend rows when reconciling fallback traffic

JEV participates in dependency authorization as typesafe/<configured model> with the evaluation role. Restricted callers need access to the dependencies they invoke. Team-member management requests cannot override the provider key or base URL. See the configuration reference for the exact settings and accounting behavior

Methodology and reproduction​

The corpus and protocol were frozen at 22:18:53 UTC, before provider calls. Measured calls ran from approximately 22:21:05 to 22:24:32 UTC. There were 20 cases in each tier, with 40 short cases, 16 long cases, eight follow-ups, eight tool-context cases and eight boundary cases. Each has an authored rationale

Both production classifier paths received the same authored rubric and opening instructions. SIMPLE covers lookup and mechanical work, MEDIUM routine drafting and localized code, COMPLEX coupled technical work, and REASONING explicit proofs, justified optimization and decisions with conflicting objectives. Their wire formats differ: the LLM path retains its trust boundary and structured-output wrapper, while JEV uses a Choice question. Neither prompt was independently tuned

A symmetric loopback HTTP relay forwarded real provider requests and retained sanitized request/response bodies without authorization headers. Timings include local relay, client, network and provider work. Haiku reported zero cache-read and cache-creation tokens. JEV did not expose cache usage, and provider-internal caching is unknown. Client and provider regions are also unknown

Quantiles use linear interpolation. The bootstrap resamples cases, keeping repeated observations together, rather than treating 240 calls as independent prompts. These intervals describe this corpus and run. They do not cover author bias, another region, sustained load, rate limits, cold starts or another account

Download the frozen raw results and reproduction scripts. The archive contains cases.jsonl, protocol.json, attempts.jsonl, sanitized wire.jsonl, metrics.json, the captured registry, environment versions, gateway evidence, setup failures and a hash manifest

curl -fL https://docs.litellm.ai/benchmarks/jev-live-evidence-20260918.tar.gz \
  -o jev-live-evidence-20260918.tar.gz
echo "c2653861ab4cf591e902d4d62d55078187e4c66ad5845b5cf5b023d5f427fd7f  jev-live-evidence-20260918.tar.gz" | sha256sum -c -
tar -xzf jev-live-evidence-20260918.tar.gz
cd jev-benchmark
sha256sum -c SHA256SUMS
python analyze.py

Offline analysis needs Python and Pydantic as listed in the archive's environment record. Its README gives the locked-repository setup and commands for fresh paid provider calls. Use a fresh run directory because the runner refuses to overwrite historical attempts. A future jev-latest resolution may produce a different model, requiring a new experiment and an explicit pricing-analysis update

The measured tree was 9dce43d6a0562a4926c39b1bacf7e9a1feac1170, reconstructed from base 1d91fc232d20a434a86323506a1303935b9c684f, budget change b9e5bb3abb0f2f0ddc06dcaf9edb63563cac2a2a first, and integration 8e5f43f45897fc72612aac53a690fa573ce029cd second. This identifies the tested combination without implying a released version

The first harness configuration combined incompatible custom-prompt fields and stopped before measured calls. Two earlier gateway configurations also used invalid or misplaced fallback settings. Their evidence is retained separately, including the first run's implicitly selected MEDIUM default. No measured case was removed or relabeled after calls began

Key Takeaways​

• JEV classified requests 5.43x as fast as Haiku by median latency, and 3.88x as fast at p95, on these 80 authored cases
• Expected-tier match rates were 95.00% versus 73.75% for this prompt set and configuration, with 78.75% agreement between classifiers. Downstream answer quality was not measured
• Registry-priced classifier cost was 96.118% lower. Total application cost and provider invoices need separate measurement
• JEV uses the existing Auto Router's context, tier pools and fallback, with a process-local timeout breaker and separate classifier spend logging
• Evaluate your own prompts and completion quality before changing a production routing policy

Frequently Asked Questions​

Does this establish LLM-quality classification?​

It establishes higher agreement with our authored labels than this Haiku configuration on this corpus. Both classifiers depend on the requests, rubric and context they receive. The labels were not independently reviewed, and repeated calls do not create new independent cases. For a broader quality claim, evaluate representative held-out traffic with independently reviewed labels, compare instruction variants, and use shadow evaluation to score the final answers

What does 5.43x as fast mean?​

Haiku's median classification time divided by JEV's was 5.43: 688.40 ms versus 126.81 ms. The title describes that measured classification speed ratio. It does not measure end-to-end completion speed or throughput under load

What happens if TypeSafe is unavailable?​

The configured fallback picks the next routing path, and recognized classification timeouts open the local circuit breaker. The chosen completion provider must still succeed. Cancellation does not establish that an upstream request was unbilled

Is JEV available in LiteLLM OSS or Enterprise?​

Built-in JEV classification is available without an Enterprise license under the same policy as the built-in LLM classifier. Custom JEV instructions and custom tier definitions use the existing Enterprise custom-classifier capability. TypeSafe provider charges are separate from LiteLLM licensing

Conclusion​

JEV classified requests 5.43x as fast as Haiku by median latency in this comparison, with lower registry-priced cost and more matches with the authored expected tiers. For Enterprise AI Gateway deployments and OSS deployments alike, the next step is to test real prompts, score the resulting answers and include fallback traffic in spend accounting. Start with the JEV setup guide and evaluate on your traffic

Recommended Reading​

• JEV Auto Router setup
• Auto Router configuration reference
• TypeSafe System One pass-through

Pricing​

$0.042 per 1M input tokens, no output charge. Spend is logged under the versioned model TypeSafe reports (typesafe/jev-1.13.0 today), even when you request jev-latest.

Usage​

Set your TypeSafe key once on the proxy, then replace https://api.typesafe.ai with LITELLM_PROXY_BASE_URL/typesafe. Clients only need a LiteLLM virtual key.

export TYPESAFE_API_KEY="your-typesafe-api-key"

curl -X POST "http://0.0.0.0:4000/typesafe/v1/systemone" \
  -H "Authorization: Bearer $LITELLM_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "state": "Help! My payouts have been failing for 3 days.",
    "model": "jev-latest",
    "questions": {
      "department": {
        "type": "choice",
        "instructions": "Which team should handle this?",
        "criteria": {
          "billing": "Payments, invoicing, refunds",
          "technical": "Bugs, outages, integrations"
        }
      }
    }
  }'

The response is TypeSafe's own, unchanged. Any path under /typesafe/ is forwarded, so GET /typesafe/v1/models lists the available models. Full details in the TypeSafe pass-through docs.

Feedback​

Running Jev through LiteLLM and hitting something unexpected? Open a GitHub discussion.

LiteLLM now keeps the readable history and removes encrypted reasoning that the newly selected tier cannot verify. The request can continue to the selected model instead of failing with invalid_encrypted_content

This fix is included in the LiteLLM v1.102.x release line and landed in PR #40280

Why encrypted reasoning blocks fail after a tier change​

A Responses API reasoning item can contain an encrypted_content field and an item ID such as rs_.... The provider binds that payload to the organization, API key, or encryption boundary that created it. A later request that carries the item to a different boundary can reach the provider, but the provider cannot verify it

Auto-Router makes this situation more common than a fixed model route. One turn may go to a fast model in the SIMPLE tier. The next turn may classify as COMPLEX and select a different model group. If the follow-up includes reasoning from the first turn, forwarding the encrypted payload to the second group produces the upstream error

Broad session or deployment affinity avoids the error by pinning more traffic than the conversation requires. It also prevents Auto-Router from switching tiers when the request needs a different model

The routing decision​

Enable the encrypted_content_affinity pre-call check alongside your Auto-Router configuration

router_settings:
  optional_pre_call_checks:
    - encrypted_content_affinity

model_list:
  - model_name: sonnet-standard
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: opus
    litellm_params:
      model: anthropic/claude-opus-5
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: production-auto-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        classifier_type: heuristic
        tiers:
          SIMPLE:
            - model_name: sonnet-standard
          COMPLEX:
            - model_name: opus

When LiteLLM sees a tracked encrypted item, it first looks for the deployment that created it. If that deployment is available, LiteLLM pins this request to it. If another healthy deployment shares the same encryption boundary, LiteLLM can use that peer

If the originating deployment belongs to a different routed group, LiteLLM removes the undecryptable encrypted reasoning and returns the healthy deployments for the selected group. Auto-Router keeps its tier decision

A same-group deployment that is temporarily unavailable follows the existing fail-fast behavior. LiteLLM does not send the encrypted item to a same-group deployment that cannot serve it and does not hide a cooldown behind a cross-tier rewrite

What LiteLLM removes​

LiteLLM mutates the request before dispatch and removes encrypted_content and id from a reasoning item when the selected deployment cannot decrypt them. It keeps a readable summary or content field so the selected model still sees the response history

If a reasoning item has no readable summary or content, LiteLLM removes that item. Other input items, including user messages and ordinary assistant messages, remain in the request

For example, a cross-tier follow-up changes from this:

{
  "type": "reasoning",
  "id": "rs_previous_deployment_item",
  "encrypted_content": "opaque-provider-payload",
  "summary": [
    {"type": "summary_text", "text": "The request needs a larger model"}
  ]
}

to this before dispatch:

{
  "type": "reasoning",
  "summary": [
    {"type": "summary_text", "text": "The request needs a larger model"}
  ]
}