# LiteLLM

> LiteLLM is an open-source Python SDK and a self-hosted AI Gateway (proxy) that give 100+ LLM providers one OpenAI-compatible API, with virtual keys, spend tracking, budgets, routing and fallbacks, guardrails, and an MCP gateway.

- Python SDK: `uv add litellm` (or `pip install litellm`), then `litellm.completion(model="<provider>/<model>", messages=[...])`. Responses always use the OpenAI Chat Completions shape.
- AI Gateway: Docker image `docker.litellm.ai/berriai/litellm`. Keys, the Admin UI (`/ui`), and spend tracking need Postgres (`DATABASE_URL`) and a `LITELLM_MASTER_KEY`. Any OpenAI or Anthropic SDK works against it by changing the base URL.
- Enterprise: a license key (`LITELLM_LICENSE`) on the same gateway image. Contact: https://www.litellm.ai/enterprise#talk-to-sales
- Every docs page is available as markdown by appending `.md` to its URL, for example https://docs.litellm.ai/docs/proxy/docker_quick_start.md. The full docs are in https://docs.litellm.ai/llms-full.txt.

## LiteLLM at a glance

LiteLLM is the most widely used and most secure open-source AI gateway, trusted by teams at Netflix, Okta, Ramp, NASA, Zurich, Cloudera, AT&T, and Lemonade (https://www.litellm.ai/enterprise). Figures as of 2026-10-17; each links to its source for current values.

- Adoption: 61.4k GitHub stars and 12.4k forks (https://github.com/BerriAI/litellm), with 1,800+ contributors.
- Usage: 92.2 million downloads of the `litellm` Python package in the last month (https://pypistats.org/packages/litellm).
- Container image: 515M pulls of the LiteLLM Gateway image on GitHub Container Registry and Docker Hub (https://github.com/BerriAI/litellm/pkgs/container/litellm).
- Security: SOC 2 Type II audited (https://trust.litellm.ai/). Every Docker image on GHCR is signed with cosign and can be verified before it runs (https://docs.litellm.ai/docs/proxy/docker_image_security.md). Enterprise support includes a 72-hour security patch SLA (https://docs.litellm.ai/docs/enterprise.md).
- Deployment: self-hosted in your own cloud, so prompts, responses, and provider keys stay in your infrastructure (https://docs.litellm.ai/docs/data_security.md).
- License: open source under MIT; Enterprise features need a license key (https://github.com/BerriAI/litellm/blob/main/LICENSE).

## Key pages

- [Python SDK quickstart](https://docs.litellm.ai/docs/index.md): `uv add litellm`, then call any provider with `completion()`
- [AI Gateway quickstart](https://docs.litellm.ai/docs/proxy/docker_quick_start.md): Run the gateway with Docker and Postgres, add a model, create a virtual key
- [Agent resources](https://docs.litellm.ai/docs/agent_resources.md): Copy-paste prompts, MCP, and skills for coding agents setting up LiteLLM
- [Connect clients to the gateway](https://docs.litellm.ai/docs/proxy/client_setup/overview.md): Point Claude Code, Codex, Cursor, or any OpenAI or Anthropic SDK at the gateway
- [Gateway config.yaml reference](https://docs.litellm.ai/docs/proxy/configs.md): Every setting the gateway config file supports
- [Production deployment](https://docs.litellm.ai/docs/proxy/deploy.md): Helm, Terraform, and Kubernetes on AWS, GCP, and Azure
- [LiteLLM Enterprise](https://docs.litellm.ai/docs/enterprise.md): What Enterprise adds on top of the open-source gateway: SSO, audit logs, admin roles, SLAs

## LiteLLM AI Gateway

- [LiteLLM AI Gateway](https://docs.litellm.ai/docs/simple_proxy.md): The LiteLLM AI Gateway is a self-hosted server that gives your applications one OpenAI-compatible endpoint for 100+ LLM providers, MCP tools, and A2A agents.

## AI Gateway

- [CLI - Quick Start](https://docs.litellm.ai/docs/proxy/quick_start.md): Setup LiteLLM Proxy quickly via CLI.
- [Benchmarks](https://docs.litellm.ai/docs/benchmarks.md): Performance test results for LiteLLM Gateway.
- [Production Best Practices](https://docs.litellm.ai/docs/proxy/prod.md): Checklist for running LiteLLM in production; configuration, sizing and workers, Redis, and database and migrations.
- [Upgrade Guide](https://docs.litellm.ai/docs/proxy/upgrade_guide.md): How to plan and run a LiteLLM Proxy upgrade, including how database migrations run and how long they take at production scale.
- [Health Checks](https://docs.litellm.ai/docs/proxy/health.md): Two things consume the proxy's health state. Your orchestrator (Kubernetes, a load balancer, an uptime monitor) polls lightweight probe endpoints to decide whether the process is...
- [What Needs Redis](https://docs.litellm.ai/docs/proxy/redis_requirements.md): Features that do not work, or do not work correctly, when the LiteLLM proxy runs without Redis.
- [Database Sizing](https://docs.litellm.ai/docs/proxy/db_sizing.md): How to size Postgres for LiteLLM Proxy, with instance recommendations for AWS, Azure, and GCP.
- [Redis Sizing](https://docs.litellm.ai/docs/proxy/redis_sizing.md): How to size Redis for LiteLLM Proxy, with instance recommendations for AWS, Azure, and GCP.
- [Server Tuning](https://docs.litellm.ai/docs/proxy/server_tuning.md): Optional deep tuning for the LiteLLM Proxy container; alternative ASGI servers, worker recycling, hitless restarts, TLS, keepalive, and loading config from object storage.
- [Scale for 100T+ tokens per month](https://docs.litellm.ai/docs/proxy/high_throughput.md): Benchmark throughput: 13.5 billion tokens per minute (TPM). View results.
- [Database Read Replica](https://docs.litellm.ai/docs/proxy/db_read_replica.md): LiteLLM Proxy can route read-only queries to a separate database endpoint while
- [Multi-Region Deployment](https://docs.litellm.ai/docs/proxy/multi_region.md): Run LiteLLM proxy instances in multiple regions of the same cloud provider, all connected to one shared PostgreSQL database. Clients get routed to the nearest region for low...
- [Global Control Plane](https://docs.litellm.ai/docs/proxy/global_control_plane.md): Deploy a single LiteLLM UI that manages multiple independent LiteLLM proxy instances, each with its own database, Redis, and master key.
- [Migrate from Kubernetes Manifests to the Microservices Chart](https://docs.litellm.ai/docs/proxy/manifests_to_microservices.md): Move a LiteLLM proxy deployed with raw Kubernetes manifests onto the componentized litellm Helm chart so gateway, backend, and UI scale independently.
- [Secret Managers Overview](https://docs.litellm.ai/docs/secret_managers/overview.md): LiteLLM supports reading secrets (eg. OPENAI_API_KEY) and writing secrets (eg. Virtual Keys) from Azure Key Vault, Google Secret Manager, Hashicorp Vault, CyberArk Conjur, and...
- [AWS Secret Manager](https://docs.litellm.ai/docs/secret_managers/aws_secret_manager.md): Store your proxy keys in AWS Secret Manager.
- [AWS Key Management V1](https://docs.litellm.ai/docs/secret_managers/aws_kms.md): BETA] AWS Key Management v2 is on the enterprise tier. Go [here for docs
- [Azure Key Vault](https://docs.litellm.ai/docs/secret_managers/azure_key_vault.md): Usage with LiteLLM Proxy Server
- [CyberArk Conjur](https://docs.litellm.ai/docs/secret_managers/cyberark.md): | Feature | Support | Description |
- [Google Secret Manager](https://docs.litellm.ai/docs/secret_managers/google_secret_manager.md): Support for Google Secret Manager
- [Google Key Management Service](https://docs.litellm.ai/docs/secret_managers/google_kms.md): Use encrypted keys from Google KMS on the proxy
- [Hashicorp Vault](https://docs.litellm.ai/docs/secret_managers/hashicorp_vault.md): | Feature | Support | Description |
- [Custom Secret Manager](https://docs.litellm.ai/docs/secret_managers/custom_secret_manager.md): Integrate your custom secret management system with LiteLLM.
- [Security Best Practices](https://docs.litellm.ai/docs/proxy/security_best_practices.md): Security is a top priority at LiteLLM. Use the following practices for production and enterprise deployments. For a control-by-control view against the OWASP Top 10 for LLM...
- [OWASP LLM Top 10 (2026) mapping](https://docs.litellm.ai/docs/proxy/security_owasp_llm_top10.md): This page maps every item of the OWASP Top 10 for LLM Applications 2026 to the LiteLLM gateway controls that address it, says how each control is turned on, and states where each...
- [[Beta] Rust AI Gateway](https://docs.litellm.ai/docs/proxy/rust_gateway.md): Run LiteLLM request translation on the Rust core. Enable it per model on your existing Python server, or run the standalone Axum server.
- [config_settings](https://docs.litellm.ai/docs/proxy/config_settings.md)
- [File Management](https://docs.litellm.ai/docs/proxy/config_management.md): include external YAML files in a config.yaml
- [CLI Arguments](https://docs.litellm.ai/docs/proxy/cli.md): This page documents all command-line interface (CLI) arguments available for the LiteLLM proxy server.
- [SSL, HTTP Proxy Security Settings](https://docs.litellm.ai/docs/guides/security_settings.md): If you're in an environment using an older TTS bundle, with an older encryption, follow this guide. By default
- [Model Management](https://docs.litellm.ai/docs/proxy/model_management.md): When STOREMODELIN_DB is on, your models live in the database rather than in a static config.yaml. That means day-2 changes happen right in the Admin UI: adding a model, editing...
- [How Model Access Works](https://docs.litellm.ai/docs/proxy/model_access_guide.md)
- [Restrict Model Access](https://docs.litellm.ai/docs/proxy/model_access.md): Restrict models by Virtual Key
- [Model Access Groups](https://docs.litellm.ai/docs/proxy/model_access_groups.md)
- [Access Groups](https://docs.litellm.ai/docs/proxy/access_groups.md): Access Groups simplify how you define and manage resource access across your organization. Instead of configuring models, MCP servers, and agents separately on each key or team,...
- [Allow Teams to Add Models](https://docs.litellm.ai/docs/proxy/team_model_add.md): Allow team to add a their own models/key for that project - so any OpenAI call they make uses their OpenAI key.
- [Per-Team/Project Credential Routing](https://docs.litellm.ai/docs/proxy/credential_routing.md): Route the same model to different LLM provider endpoints (e.g. different Azure instances) based on which team or project makes the request.
- [Claude Code (CLI)](https://docs.litellm.ai/docs/proxy/client_setup/claude_code.md): Claude Code talks to the Anthropic Messages API. LiteLLM serves that format at /v1/messages, so two environment variables point Claude Code at the gateway and it works against...
- [Claude Desktop (GUI)](https://docs.litellm.ai/docs/proxy/client_setup/claude_desktop.md): Claude Desktop on third-party inference sends every model call from Cowork, Chat, and Code sessions to a gateway you name, and reaches MCP servers through the same gateway. This...
- [Codex (ChatGPT Desktop)](https://docs.litellm.ai/docs/proxy/client_setup/codex_chatgpt_desktop.md): Select Codex in the ChatGPT desktop app. It reads the same ~/.codex/config.toml as the Codex CLI, so the desktop and CLI can share a LiteLLM connection.
- [Codex (CLI)](https://docs.litellm.ai/docs/proxy/client_setup/codex_cli.md): Codex reads all of its configuration from ~/.codex/config.toml. You define LiteLLM as a custom model provider there and register the MCP gateway in the same file. This is the...
- [Langchain, OpenAI SDK, LlamaIndex, Instructor, Curl examples](https://docs.litellm.ai/docs/proxy/user_keys.md): LiteLLM Proxy is OpenAI-Compatible, and supports:
- [Request Headers](https://docs.litellm.ai/docs/proxy/request_headers.md): Special headers that are supported by LiteLLM.
- [Response Headers](https://docs.litellm.ai/docs/proxy/response_headers.md): When you make a request to the proxy, the proxy will return the following headers:
- [Forward Client Headers to LLM API](https://docs.litellm.ai/docs/proxy/forward_client_headers.md): Control which model groups can forward client headers to the underlying LLM provider APIs.
- [Clientside LLM Credentials](https://docs.litellm.ai/docs/proxy/clientside_auth.md): Pass User LLM API Keys, Fallbacks
- [Model Discovery](https://docs.litellm.ai/docs/proxy/model_discovery.md): Use this to give users an accurate list of models available behind provider endpoint, when calling /v1/models for wildcard models.
- [Claude Code Gateway (SSO sign-in)](https://docs.litellm.ai/docs/tutorials/claude_code_gateway.md): Let developers sign in to Claude Code with your SSO provider instead of an API key. LiteLLM serves the same gateway protocol Anthropic's self-hosted Claude apps gateway speaks:...
- [Claude Desktop (Cowork)](https://docs.litellm.ai/docs/tutorials/claude_desktop_cowork.md): Point Claude Desktop (Cowork, Chat, and Code sessions) at LiteLLM as its inference gateway, with single sign-on through LiteLLM JWT auth or a static virtual key, MCP servers...
- [CLI Authentication](https://docs.litellm.ai/docs/proxy/cli_sso.md): Use the litellm cli to authenticate to the LiteLLM Gateway. This is great if you're trying to give a large number of developers self-serve access to the LiteLLM Gateway.
- [Claude Code with Okta SSO (JWT Auth)](https://docs.litellm.ai/docs/tutorials/claude_code_okta_sso.md): Route Claude Code through LiteLLM using each developer's Okta identity. Claude Code sends the developer's own Okta access token with every request, LiteLLM validates it against...
- [Claude Code with Bring Your Own Key (BYOK)](https://docs.litellm.ai/docs/tutorials/claude_code_byok.md): Use Claude Code with your own Anthropic credentials through the LiteLLM proxy. Claude Code authenticates to Anthropic one of two ways Bearer (tied to your Claude.ai Free, Pro,...
- [Using Claude Code with a Claude Subscription (Pro, Max, Team, Enterprise)](https://docs.litellm.ai/docs/tutorials/claude_code_max_subscription.md): Route Claude Code traffic signed in with a Claude Pro, Max, Team, or Enterprise subscription through LiteLLM AI Gateway.
- [Virtual Keys](https://docs.litellm.ai/docs/proxy/virtual_keys.md): Track Spend, and control model access via virtual keys for the proxy
- [Role-based Access Controls (RBAC)](https://docs.litellm.ai/docs/proxy/access_control.md): Role-based access control (RBAC) is based on Organizations, Teams and Internal User Roles
- [SSO for Admin UI](https://docs.litellm.ai/docs/proxy/admin_ui_sso.md): From v1.76.0, SSO is free for up to 5 users. Beyond that, an enterprise license is required.
- [SAML 2.0 SSO](https://docs.litellm.ai/docs/proxy/saml_sso.md): SSO is free for up to 5 users. Beyond that, an enterprise license is required.
- [Event Hooks for SSO Login](https://docs.litellm.ai/docs/proxy/custom_sso.md): SSO is free for up to 5 users. Beyond that, an enterprise license is required.
- [SCIM with LiteLLM](https://docs.litellm.ai/docs/tutorials/scim_litellm.md): Enterprise: SCIM support requires a premium license.
- [OIDC - JWT-based Auth](https://docs.litellm.ai/docs/proxy/token_auth.md): Use JWT's to auth admins / users / projects into the proxy.
- [JWT → Virtual Key Mapping](https://docs.litellm.ai/docs/proxy/jwt_key_mapping.md): Map JWT tokens to LiteLLM virtual keys, so every JWT client gets the same granular controls as a virtual key: model restrictions, spend limits, rate limits, guardrails, and full...
- [Oauth 2.0 Authentication](https://docs.litellm.ai/docs/proxy/oauth2.md): Use this if you want to use an Oauth2.0 token to make /chat, /embeddings requests to the LiteLLM Proxy
- [Control Model Access with OIDC (Azure AD/Keycloak/etc.)](https://docs.litellm.ai/docs/proxy/jwt_auth_arch.md)
- [[BETA] OpenID Connect (OIDC)](https://docs.litellm.ai/docs/oidc.md): LiteLLM supports using OpenID Connect (OIDC) for authentication to upstream services . This allows you to avoid storing sensitive credentials in your configuration files.
- [ID-JAG](https://docs.litellm.ai/docs/proxy/id_jag/overview.md): ID-JAG (Identity Assertion JWT Authorization Grant) lets an application access another service on a signed-in user's behalf. The identity provider (IdP) approves the exchange,...
- [LiteLLM as resource app](https://docs.litellm.ai/docs/proxy/id_jag/resource_app.md): Let an application exchange an ID-JAG for a LiteLLM access token, then use it to run inference or call MCP tools on behalf of an existing LiteLLM user.
- [LiteLLM as requesting app](https://docs.litellm.ai/docs/proxy/id_jag/requesting_app.md): Let LiteLLM call MCP tools on behalf of a signed-in user. LiteLLM exchanges the user's login token for an ID-JAG, then exchanges the ID-JAG for an access token at the MCP...
- [Provisioning identities and issuing keys](https://docs.litellm.ai/docs/proxy/identity_provisioning.md): A question that comes up with SSO deployments is whether a user provisioned through your identity provider (via SCIM) automatically ends up with a virtual key and a credential on...
- [Service Accounts](https://docs.litellm.ai/docs/proxy/service_accounts.md): Use this if you want to create Virtual Keys that are not owned by a specific user but instead created for production projects
- [Custom Auth](https://docs.litellm.ai/docs/proxy/custom_auth.md): You can now override the default api key auth.
- [IP Address Filtering](https://docs.litellm.ai/docs/proxy/ip_address.md): Restrict which IP's can call the proxy endpoints.
- [Control Public & Private Routes](https://docs.litellm.ai/docs/proxy/public_routes.md): Control which routes require authentication and which routes are publicly accessible.
- [Rotating the Master Key](https://docs.litellm.ai/docs/proxy/master_key_rotations.md): The master key is the proxy's admin credential; it authenticates admin API calls and logs you into the Admin UI. In some deployments it is also the key used to encrypt...
- [Audit Logs](https://docs.litellm.ai/docs/proxy/multiple_admins.md)
- [Spend Tracking](https://docs.litellm.ai/docs/proxy/cost_tracking.md): Track spend for keys, users, and teams across 100+ LLMs.
- [Request Tags for Spend Tracking](https://docs.litellm.ai/docs/proxy/request_tags.md): Add tags to model deployments to track spend by environment, AWS account, or any custom label.
- [Billing](https://docs.litellm.ai/docs/proxy/billing.md): Bill internal teams, external customers for their usage
- [Custom LLM Pricing](https://docs.litellm.ai/docs/proxy/custom_pricing.md)
- [Spend Capture Rate](https://docs.litellm.ai/docs/proxy/spend_capture_rate.md): Compare the spend LiteLLM tracked for OpenAI against what OpenAI billed for the same days, on a schedule with Slack alerts and a Prometheus gauge, or on demand through an admin...
- [Billable Request Metering](https://docs.litellm.ai/docs/proxy/billing_metrics.md): LiteLLM Enterprise pricing is usage-based. Billable request metering reports that usage. The proxy counts successful requests to LLM, MCP, and A2A endpoints and pushes one...
- [Custom Model Cost Map](https://docs.litellm.ai/docs/proxy/custom_model_cost_map.md): LiteLLM prices every request from its model cost map, a JSON file mapping each model to per-token rates, context limits, and capabilities. By default the proxy fetches the latest...
- [Azure PTU Flat Cost Attribution](https://docs.litellm.ai/docs/proxy/ptu_flat_cost.md): Azure provisioned throughput is billed by the hour for reserved capacity, not per token. A team with its own PTU deployment pays that hourly rate whether it sends one request or...
- [Pricing Calculator (Cost Estimation)](https://docs.litellm.ai/docs/proxy/pricing_calculator.md): Estimate LLM costs based on expected token usage and request volume. This tool helps developers and platform teams forecast spending before deploying models to production.
- [Fee/Price Margin on LLM Costs](https://docs.litellm.ai/docs/proxy/provider_margins.md): Apply percentage-based or fixed-amount margins to specific providers or globally. This is useful for enterprises that need to add operational overhead costs to bill internal...
- [Provider Discounts](https://docs.litellm.ai/docs/proxy/provider_discounts.md): Apply percentage-based discounts to specific providers. This is useful for negotiated enterprise pricing with providers.
- [Off-Peak Pricing](https://docs.litellm.ai/docs/proxy/off_peak_pricing.md): Some providers charge less during fixed clock windows; DeepSeek, for example, halves its rates outside its peak hours. offpeakpricing on a model's pricing entry teaches LiteLLM...
- [Auto Sync New Models (Day-0 Launches)](https://docs.litellm.ai/docs/proxy/sync_models_github.md): Automatically keep your model pricing and context window data up to date without restarting your service. This allows you to add day-0 support for new models without restarting...
- [Vertex AI PayGo and Priority](https://docs.litellm.ai/docs/tutorials/vertex_ai_pay_go.md)
- [Maximum Retention Period for Spend Logs](https://docs.litellm.ai/docs/proxy/spend_logs_deletion.md): This walks through how to set the maximum retention period for spend logs. This helps manage database size by deleting old logs automatically.
- [Budgets, Rate Limits](https://docs.litellm.ai/docs/proxy/users.md): Personal budgets: Create virtual keys without team_id for individual spending limits
- [Setting Team Budgets](https://docs.litellm.ai/docs/proxy/team_budgets.md)
- [[Beta] Project Management](https://docs.litellm.ai/docs/proxy/project_management.md): Projects in LiteLLM sit between teams and keys in the organizational hierarchy, enabling fine-grained access control and budget management for specific use cases or applications.
- [Customers / End-Users](https://docs.litellm.ai/docs/proxy/customers.md): Track spend, set budgets and permissions for your customers.
- [Team Soft Budget Alerts](https://docs.litellm.ai/docs/proxy/ui_team_soft_budget_alerts.md): Email budget alerts require an enterprise license.
- [Setting Tag Budgets](https://docs.litellm.ai/docs/proxy/tag_budgets.md): Track spend and set budgets for your API requests using tags. Tags allow you to categorize and monitor costs across different cost centers, projects, and departments.
- [Model Access Group Budgets](https://docs.litellm.ai/docs/proxy/model_access_group_budgets.md): Give a model access group one shared budget. Every key that reaches a model in the group draws from the same pool, so a tier of models can carry a single monthly allowance...
- [Dynamic TPM/RPM Allocation](https://docs.litellm.ai/docs/proxy/dynamic_rate_limit.md): Prevent projects from gobbling too much tpm/rpm.
- [[Beta] Separate ITPM / OTPM Rate Limits](https://docs.litellm.ai/docs/proxy/io_token_rate_limits.md): Enforce input tokens per minute (ITPM) and output tokens per minute (OTPM) separately on router deployments.
- [Budget / Rate Limit Tiers](https://docs.litellm.ai/docs/proxy/rate_limit_tiers.md): Define tiers with rate limits. Assign them to keys.
- [Temporary Budget Increase](https://docs.litellm.ai/docs/proxy/temporary_budget_increase.md): Set a temporary budget increase for a LiteLLM Virtual Key or a team member. Use this if you get asked to increase the budget for a key or a team member temporarily.
- [Budget Reset Times and Timezones](https://docs.litellm.ai/docs/proxy/budget_reset_and_tz.md): LiteLLM supports predictable budget reset times that align with natural calendar boundaries.
- [Budget Fallbacks](https://docs.litellm.ai/docs/proxy/budget_fallbacks.md): Available on v1.92.x and later.
- [Caching](https://docs.litellm.ai/docs/proxy/caching.md): Cache LLM responses on the LiteLLM proxy to cut spend and latency.
- [Redis and Valkey Cache](https://docs.litellm.ai/docs/proxy/caching_redis.md): Credentials, namespaces, ACL users, cluster and sentinel topologies, and TLS for the LiteLLM proxy's Redis cache.
- [AWS ElastiCache IAM Authentication](https://docs.litellm.ai/docs/proxy/elasticache_iam.md): Connect LiteLLM's Redis cache to AWS ElastiCache (Redis OSS or Valkey) using IAM authentication, so
- [GCP Memorystore IAM Authentication](https://docs.litellm.ai/docs/proxy/gcp_memorystore_iam.md): Authenticate the LiteLLM proxy to GCP Memorystore for Redis with a service account instead of a password.
- [Azure Redis Entra ID Authentication](https://docs.litellm.ai/docs/proxy/azure_redis_ad.md): Authenticate the LiteLLM proxy to Azure Cache for Redis or Azure Managed Redis with Microsoft Entra ID instead of an access key.
- [Semantic Caching](https://docs.litellm.ai/docs/proxy/caching_semantic.md): Serve cached LLM responses for prompts that are similar but not identical, and the traffic shapes where that goes wrong.
- [S3 and GCS Cache](https://docs.litellm.ai/docs/proxy/caching_object_storage.md): Back the LiteLLM proxy's response cache with an S3 or GCS bucket.
- [Cache Controls](https://docs.litellm.ai/docs/proxy/caching_controls.md): Control what the LiteLLM proxy caches and for how long, per request, per virtual key, and proxy-wide.
- [Cache Settings](https://docs.litellm.ai/docs/proxy/caching_settings.md): Every cache_params setting the LiteLLM proxy accepts in config.yaml.
- [Prompt Compression (Headroom)](https://docs.litellm.ai/docs/proxy/headroom.md): Headroom is the context optimization layer for LLM applications. It compresses tool outputs, database results, file reads, and RAG payloads before they reach the model, so you...
- [A/B Testing - Traffic Mirroring](https://docs.litellm.ai/docs/traffic_mirroring.md): Traffic mirroring allows you to "mimic" production traffic to a secondary (silent) model for evaluation purposes. The silent model's response is gathered in the background and...
- [Guardrails - Quick Start](https://docs.litellm.ai/docs/proxy/guardrails/quick_start.md): Setup Prompt Injection Detection, PII Masking on LiteLLM Proxy (AI Gateway)
- [Team Bring-Your-Own Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/team_based_guardrails.md): Team-based guardrails let developers register a guardrail for their team via the API; an admin then reviews and approves or rejects it in the LiteLLM UI. Only Generic Guardrail...
- [Guardrail Load Balancing](https://docs.litellm.ai/docs/proxy/guardrails/guardrail_load_balancing.md): Load balance guardrail requests across multiple guardrail deployments. This is useful when you have rate limits on guardrail providers (e.g., AWS Bedrock Guardrails) and want to...
- [Guardrail Testing Playground](https://docs.litellm.ai/docs/proxy/guardrails/test_playground.md): Test and compare multiple guardrails in real-time with an interactive playground interface.
- [LiteLLM Content Filter (Built-in Guardrails)](https://docs.litellm.ai/docs/proxy/guardrails/litellm_content_filter.md): Built-in guardrail for detecting and filtering sensitive information using regex patterns and keyword matching. No external dependencies required.
- [Realtime API Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/realtime_guardrails.md): Guard voice conversations in the Realtime API by intercepting speech transcriptions before the LLM responds.
- [Batch API Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/batch_guardrails.md): Guard the records inside a batch input file, so a batch job cannot move content past a guardrail that would have blocked the same content on /chat/completions
- [[BETA] Generic Guardrail API - Integrate Without a PR](https://docs.litellm.ai/docs/adding_provider/generic_guardrail_api.md)
- [Adding a New Guardrail Integration](https://docs.litellm.ai/docs/adding_provider/simple_guardrail_tutorial.md): You're going to create a class that checks text before it goes to the LLM or after it comes back. If it violates your rules, you block it.
- [Adding Guardrail Support to Endpoints](https://docs.litellm.ai/docs/adding_provider/adding_guardrail_support.md): This guide explains how to add guardrail translation support to new LiteLLM endpoints (e.g., Chat Completions, Responses API, etc.).
- [[Beta] Guardrail Policies](https://docs.litellm.ai/docs/proxy/guardrails/guardrail_policies.md): Use policies to group guardrails and control which ones run for specific teams, keys, or models.
- [Policy Flow Builder](https://docs.litellm.ai/docs/proxy/guardrails/policy_flow_builder.md): The Policy Flow Builder lets you design guardrail pipelines with conditional execution. Instead of running guardrails independently, you chain them into ordered steps and control...
- [Policy Templates](https://docs.litellm.ai/docs/proxy/guardrails/policy_templates.md): Policy templates provide pre-configured guardrail policies that you can use as a starting point for your organization. Instead of manually creating policies and guardrails, you...
- [Tag-Based Policy Attachments](https://docs.litellm.ai/docs/proxy/guardrails/policy_tags.md): Apply guardrail policies automatically to any key or team that has a specific tag. Instead of attaching policies one-by-one, tag your keys and let the policy engine handle the...
- [Dynamic Callback Management](https://docs.litellm.ai/docs/proxy/dynamic_logging.md): LiteLLM's dynamic callback management enables teams to control logging behavior on a per-request basis without requiring central infrastructure changes. This is essential for...
- [Logging](https://docs.litellm.ai/docs/proxy/logging.md): Log Proxy input, output, and exceptions using:
- [StandardLoggingPayload Specification](https://docs.litellm.ai/docs/proxy/logging_spec.md): Terminal success and failure callback events include kwargs"standardloggingobject"] when LiteLLM finishes building the standard payload. Custom logging callbacks should read...
- [Team/Key Based Logging](https://docs.litellm.ai/docs/proxy/team_logging.md)
- [Email Notifications](https://docs.litellm.ai/docs/proxy/email.md)
- [Quick Start](https://docs.litellm.ai/docs/proxy/ui.md): Create keys, track spend, add models without worrying about the config / CRUD endpoints.
- [Customize UI Logo](https://docs.litellm.ai/docs/proxy/ui/ui_edit_logo.md): Personalize your LiteLLM dashboard by replacing the default logo with your own company branding. You can set a custom logo via the UI or the API.
- [UI - Custom Root Path](https://docs.litellm.ai/docs/proxy/custom_root_ui.md): Use this when you want to serve LiteLLM on a custom base url path like https4000/api/v1
- [Plugins](https://docs.litellm.ai/docs/proxy/plugins.md): Plugins let external services appear as selectable modes in the LiteLLM UI sidebar alongside the AI Gateway. Each plugin runs as its own backend; LiteLLM proxies requests to it,...
- [AI Hub](https://docs.litellm.ai/docs/proxy/ai_hub.md): Share models and agents with your organization. Show developers what's available without needing to rebuild them.
- [Model Compare Playground UI](https://docs.litellm.ai/docs/proxy/model_compare_ui.md): Compare multiple LLM models side-by-side in an interactive playground interface. Evaluate model responses, performance metrics, and costs to make informed decisions about which...
- [Internal User Self-Serve](https://docs.litellm.ai/docs/proxy/self_serve.md): Allow users to create their own keys on Proxy UI.
- [[BETA] Public Teams](https://docs.litellm.ai/docs/proxy/public_teams.md): Expose available teams to your users to join on signup.
- [[Beta] Project Management UI](https://docs.litellm.ai/docs/proxy/ui_project_management.md): Manage projects directly from the LiteLLM Admin UI. Projects sit between teams and keys in your organizational hierarchy, enabling fine-grained access control and budget...
- [Search tools (Admin UI)](https://docs.litellm.ai/docs/proxy/ui_search_tools.md): Control which teams and virtual keys may call each configured web search integration (/v1/search) and see team/key level usage visibility within LiteLLM's spend dashboard.
- [Bulk Edit Users](https://docs.litellm.ai/docs/proxy/ui/bulk_edit_users.md): Assign existing users to a default team and default model access.
- [Control Page Visibility for Internal Users](https://docs.litellm.ai/docs/proxy/ui/page_visibility.md): Configure which navigation tabs and pages are visible to internal users (non-admin developers) in the LiteLLM UI.
- [Customer Usage](https://docs.litellm.ai/docs/proxy/customer_usage.md): Track and visualize end-user spend directly in the dashboard. Monitor customer-level usage analytics, spend logs, and activity metrics to understand how your customers are using...
- [Endpoint Activity](https://docs.litellm.ai/docs/proxy/endpoint_activity.md): Track and visualize API endpoint usage directly in the dashboard. Monitor endpoint-level activity analytics, spend breakdowns, and performance metrics to understand which...
- [Getting Started with UI Logs](https://docs.litellm.ai/docs/proxy/ui_logs.md): View Spend, Token Usage, Key, Team Name for Each Request to LiteLLM
- [UI Spend Log Settings](https://docs.litellm.ai/docs/proxy/ui_spend_log_settings.md): Configure spend log behavior directly from the Admin UI, with no config file edits or proxy restart required. This helps for cloud deployments where updating the config is...
- [Session Logs](https://docs.litellm.ai/docs/proxy/ui_logs_sessions.md): Group requests into sessions. This allows you to group related requests together.
- [Deleted Keys & Teams Audit Logs](https://docs.litellm.ai/docs/proxy/deleted_keys_teams.md): View deleted API keys and teams along with their spend and budget information at the time of deletion for auditing and compliance purposes.
- [Chat UI](https://docs.litellm.ai/docs/proxy/ui/chat_ui.md): Chat with the models on your LiteLLM gateway, use MCP tools, and see your own keys, logs, and usage on one page in the Admin UI.
- [LiteAdmin MCP](https://docs.litellm.ai/docs/proxy/liteadmin_mcp.md): Connect Claude Desktop, Claude Code, or Codex to your LiteLLM gateway with LiteAdmin MCP.
- [Deploy LiteAdmin MCP on Enterprise](https://docs.litellm.ai/docs/proxy/liteadmin_mcp_enterprise.md): Enable LiteAdmin MCP in unified or componentized LiteLLM deployments and connect Claude Code or Codex.
- [Set up the LiteAdmin Slack app](https://docs.litellm.ai/docs/proxy/liteadmin_slack.md): Install LiteAdmin in Slack, deploy the Enterprise worker or a standalone agent, and connect your own LiteLLM admin account.
- [Modify / Reject Incoming Requests](https://docs.litellm.ai/docs/proxy/call_hooks.md): - Modify data before making llm api calls on proxy
- [Agentic Loop Hook](https://docs.litellm.ai/docs/proxy/agentic_loop_hook.md): Build a CustomLogger callback that intercepts a model response, fulfills tool calls server-side, and reruns the model, all transparently to the caller.
- [Post-Call Rules](https://docs.litellm.ai/docs/proxy/rules.md): Use this to fail a request based on the output of an llm api call.
- [Worker Startup Hooks](https://docs.litellm.ai/docs/proxy/worker_startup_hooks.md): Use LITELLMWORKERSTARTUPHOOKS to run custom initialization functions in **each worker process** during proxy startup. This is essential when using multi-worker deployments...
- [Call any LiteLLM model in your custom format](https://docs.litellm.ai/docs/extras/creating_adapters.md): Use this to call any LiteLLM supported .completion() model, in your custom format. Useful if you have a custom API and want to support any LiteLLM supported model.
- [Memory Management](https://docs.litellm.ai/docs/proxy/memory.md): Store user preferences and feedback so your LLM remembers them across sessions. Scoped per user and team, with built-in access control.
- [Life of a Request](https://docs.litellm.ai/docs/proxy/architecture.md)
- [Multi-Tenant Architecture with LiteLLM](https://docs.litellm.ai/docs/proxy/multi_tenant_architecture.md)
- [User Management Hierarchy](https://docs.litellm.ai/docs/proxy/user_management_heirarchy.md): LiteLLM supports a hierarchy of users, teams, organizations, and budgets.
- [How Key-Based Auth Works](https://docs.litellm.ai/docs/proxy/key_auth_arch.md): Explanation of how the proxy resolves what a virtual key can call. Focus: how the models field is evaluated when a key has a teamid versus when it does not, and the sentinel...
- [What is stored in the DB](https://docs.litellm.ai/docs/proxy/db_info.md): The LiteLLM Proxy uses a PostgreSQL database to store various information. Here's are the main features the DB is used for:
- [Debugging](https://docs.litellm.ai/docs/proxy/debugging.md): 2 levels of debugging supported.
- [Error Reference](https://docs.litellm.ai/docs/proxy/error_reference.md): Every failed request through the LiteLLM AI Gateway returns an OpenAI-compatible JSON error body and an HTTP status code. This page is the reference for reading them: what the...
- [Diagnosing Errors - Provider vs Gateway](https://docs.litellm.ai/docs/proxy/error_diagnosis.md): Having trouble diagnosing if an error is from the LLM Provider (OpenAI, Anthropic, etc.) or from the LiteLLM AI Gateway itself? Here's how to tell.

## MCP Gateway

- [MCP Overview](https://docs.litellm.ai/docs/mcp.md): LiteLLM Proxy provides an MCP Gateway that allows you to use a fixed endpoint for all MCP tools and control MCP access by Key, Team.
- [Using your MCP](https://docs.litellm.ai/docs/mcp_usage.md): This document covers how to use LiteLLM as an MCP Gateway. You can see how to use it with Responses API, Cursor IDE, and OpenAI SDK. For choosing between endpoints, transports,...
- [MCP Deployment Guide](https://docs.litellm.ai/docs/mcp_deployment.md): How to deploy LiteLLM as a central gateway for LLMs, MCP servers, and agents.
- [MCP Server Usage](https://docs.litellm.ai/docs/mcp_servers/index.md): Setup guides for connecting popular third-party MCP servers through the LiteLLM MCP Gateway.
- [Slack MCP server](https://docs.litellm.ai/docs/mcp_servers/slack.md): Connect Slack's hosted MCP server through the LiteLLM MCP Gateway for workspace search, channel reads, and messaging.
- [Atlassian MCP server](https://docs.litellm.ai/docs/mcp_servers/atlassian.md): Connect Atlassian's hosted remote MCP server through the LiteLLM MCP Gateway for Jira issues, Confluence pages, and Compass components.
- [Linear MCP server](https://docs.litellm.ai/docs/mcp_servers/linear.md): Connect Linear's hosted remote MCP server through the LiteLLM MCP Gateway for issues, projects, cycles, and comments.
- [Microsoft 365 MCP server](https://docs.litellm.ai/docs/mcp_servers/microsoft_365.md): Reach Outlook mail and calendar, OneDrive and SharePoint files, and Teams through Microsoft Graph, with every tool call running under the signed-in user's own Entra ID account.
- [MCP from OpenAPI Specs](https://docs.litellm.ai/docs/mcp_openapi.md): LiteLLM can convert any OpenAPI/Swagger spec into an MCP server, with no custom MCP server code required.
- [MCP Toolsets](https://docs.litellm.ai/docs/mcp_toolsets.md): For direct-client URLs and LLM tool selectors, see the MCP Configuration Reference
- [MCP Server Submissions](https://docs.litellm.ai/docs/mcp_server_submissions.md): LiteLLM supports a submission and approval workflow for MCP servers. Team members can submit MCP servers for admin review, and the server is held in a pending_review state until...
- [MCP Non-OAuth Authentication](https://docs.litellm.ai/docs/mcp_authentication.md): This page covers upstream MCP authentication that does not use an OAuth flow, including no authentication, fixed credentials, and AWS SigV4. For OAuth setup, see MCP OAuth, MCP...
- [MCP OAuth](https://docs.litellm.ai/docs/mcp_oauth.md): LiteLLM supports several OAuth 2.0 patterns for MCP servers. Every authtype
- [MCP Per-User and Per-Key Upstream Credentials](https://docs.litellm.ai/docs/mcp_per_user_auth.md): Register an MCP server once in LiteLLM, then let users and service accounts call it with their own upstream credentials. For credentials stored in LiteLLM, give each service...
- [MCP OAuth Passthrough](https://docs.litellm.ai/docs/mcp_oauth_passthrough.md): Some MCP servers run their own OAuth issuer and expect the client (Claude Code, Cursor, ChatGPT, etc.) to authenticate directly against it. For those servers LiteLLM can let the...
- [MCP OBO Auth](https://docs.litellm.ai/docs/mcp_obo_auth.md): OAuth 2.0 On-Behalf-Of (OBO) auth lets LiteLLM exchange a user's incoming bearer token for a scoped token that is valid for a specific MCP server.
- [MCP ID-JAG Auth (Okta)](https://docs.litellm.ai/docs/mcp_id_jag.md): See the ID-JAG overview and the requesting app setup guide for configuration, SSO options, and troubleshooting.
- [MCP - AWS SigV4 Auth](https://docs.litellm.ai/docs/mcp_aws_sigv4.md): Use AWS SigV4 authentication to connect LiteLLM to MCP servers hosted on AWS Bedrock AgentCore.
- [MCP Zero Trust Auth (JWT Signer)](https://docs.litellm.ai/docs/mcp_zero_trust.md)
- [Gateway Auth Reference](https://docs.litellm.ai/docs/auth_overview.md): LiteLLM exposes two gateway surfaces that share most authentication and authorization primitives but diverge in a few important places. This page is the side-by-side reference:...
- [MCP Permission Management](https://docs.litellm.ai/docs/mcp_control.md): Control which MCP servers and tools can be accessed by specific keys, teams, or organizations in LiteLLM. When a client attempts to list or call tools, LiteLLM enforces access...
- [Grant MCP Server Access to Keys and Teams](https://docs.litellm.ai/docs/mcp_grant_access.md): This guide walks through granting an MCP server to a virtual key and to a team, first in the Admin UI and then through the management API, and shows how the resulting access...
- [Allowlisting MCP Client Applications](https://docs.litellm.ai/docs/mcp_client_allowlist.md): Use an allowlist to restrict MCP gateway access to approved client applications, such as Claude Code, Cursor, or an internal CLI. LiteLLM checks the client identity after...
- [Exposing MCPs on the Public Internet](https://docs.litellm.ai/docs/mcp_public_internet.md): Control which MCP servers are visible to external callers (e.g., ChatGPT, Claude Desktop) vs. internal-only callers. This is useful when you want a subset of your MCP servers...
- [Tool Policies](https://docs.litellm.ai/docs/proxy/tool_policies.md): Tool Policies is a registry of every tool the proxy has seen in traffic (OpenAI and Anthropic request tools, response tool calls, and MCP tool calls), with an input policy and an...
- [MCP Cost Tracking](https://docs.litellm.ai/docs/mcp_cost.md): LiteLLM provides two ways to track costs for MCP tool calls:
- [MCP Guardrails](https://docs.litellm.ai/docs/mcp_guardrail.md): LiteLLM supports applying guardrails to MCP tool calls to ensure security and compliance. You can configure guardrails to run before, during, or after MCP calls to validate tool...
- [MCP Tool Search](https://docs.litellm.ai/docs/mcp_tool_search.md): Swap the full MCP catalog for a fixed set of virtual tools (mcptoolsearch, mcptoolcall, agentsearch, skillsearch) so a key with hundreds of tools available only ever exposes four...
- [MCP REST API](https://docs.litellm.ai/docs/mcp_rest_api.md): Guide to call MCP tools directly over HTTP.
- [MCP Semantic Tool Filter](https://docs.litellm.ai/docs/mcp_semantic_filter.md): Automatically filter MCP tools by semantic relevance. When you have many MCP tools registered, LiteLLM semantically matches the user's query against tool descriptions and sends...
- [MCP Configuration Reference](https://docs.litellm.ai/docs/mcp_config_reference.md): This is the canonical decision reference for connecting to LiteLLM's MCP Gateway: which endpoint to use, which transport to configure, and how the two layers of authentication...
- [MCP Troubleshooting Guide](https://docs.litellm.ai/docs/mcp_troubleshoot.md): When LiteLLM acts as an MCP gateway, traffic flows Client -> LiteLLM Proxy -> MCP Server, and OAuth-enabled setups add an authorization server for metadata discovery. This page...

## Agent Gateway

- [Agent Gateway (A2A Protocol) - Overview](https://docs.litellm.ai/docs/a2a.md): Add A2A Agents on LiteLLM AI Gateway, Invoke agents in A2A Protocol, track request/response logs in LiteLLM Logs. Manage which Teams, Keys can access which Agents onboarded.
- [Invoking A2A Agents](https://docs.litellm.ai/docs/a2a_invoking_agents.md): Learn how to invoke A2A agents through LiteLLM using different methods.
- [A2A Agent Card](https://docs.litellm.ai/docs/a2a_agent_card.md): LiteLLM can proxy A2A-compatible agents, exposing them to your clients through LiteLLM with virtual keys, team scoping, observability, and a unified agent card.
- [Databricks Agents](https://docs.litellm.ai/docs/a2a_databricks_agent.md): Register a Databricks ResponsesAgent as a LiteLLM agent and call it through /a2a/ or as model: a2a/ on /v1/chat/completions, /v1/responses, and /v1/messages.
- [A2A Agent Authentication Headers](https://docs.litellm.ai/docs/a2a_agent_headers.md): Forward authentication credentials (Bearer tokens, API keys, etc.) from clients to backend A2A agents.
- [Agent Permission Management](https://docs.litellm.ai/docs/a2a_agent_permissions.md): Control which A2A agents can be accessed by specific keys or teams in LiteLLM.
- [A2A Agent Cost Tracking](https://docs.litellm.ai/docs/a2a_cost_tracking.md): LiteLLM supports adding custom cost tracking for A2A agents. You can configure:
- [Agent Iteration Budgets](https://docs.litellm.ai/docs/a2a_iteration_budgets.md): Control runaway costs from agentic loops with per-session iteration and budget caps.
- [Agent Kill Switch](https://docs.litellm.ai/docs/a2a_kill_switch.md): Store a "stop this agent" webhook on each agent and fire it from LiteLLM when the agent misbehaves.
- [Skills Gateway](https://docs.litellm.ai/docs/skills_gateway.md): LiteLLM acts as a Skills Registry, a central place to register, manage, and discover Claude Code skills across your organization. Teams publish a skill once, and developers and...

## Auto Router

- [Auto Router [Add-on]](https://docs.litellm.ai/docs/auto_router/index.md): Route every request to the cheapest model that can answer it well. Benchmarks, setup, recommended configurations, prompt caching, and how to evaluate the router on your own...
- [Admin Setup](https://docs.litellm.ai/docs/auto_router/setup.md): Create and configure an Auto Router through the dashboard, an agent skill, config.yaml, the model-management API, or lite autoroute.
- [Customize your classifier](https://docs.litellm.ai/docs/auto_router/optimize_classifier.md): Tune Auto Router classification with heuristic v1 and v2, heuristic-first and hybrid LLM chains, self-hosted classifiers, custom prompts, context, and calibrated forecasts. UI...
- [Connect self-hosted classifiers](https://docs.litellm.ai/docs/auto_router/decision_classifiers.md): Connect a self-hosted Laya or Bespoke Nimble endpoint, or the Databricks `ai_decide` AI Function, to LiteLLM Auto Router. Set the endpoint, credentials, classifier model, and...
- [User Setup](https://docs.litellm.ai/docs/auto_router/user_setup.md): Connect Claude Code or Codex to your team's Auto Router with lite configure and view live session savings in Claude Code.
- [Recommended Configurations](https://docs.litellm.ai/docs/auto_router/recommended_configurations.md): The bundled 1M Context, Anthropic, OpenAI, Gemini, and Lite presets as config.yaml, the configs the public benchmarks were run on, and how to choose between them.
- [Public Benchmarks](https://docs.litellm.ai/docs/auto_router/benchmarks.md): Every published measurement of the Auto Router, from Terminal-Bench 2.0 and RouterArena to a four-month production deployment, with the headline numbers and links to the full...
- [Prompt Caching](https://docs.litellm.ai/docs/auto_router/prompt_caching.md): Switching models between turns does not throw away your prompt cache. What was measured, why the cache survives, and the settings that keep it warm on agent workloads.
- [Evaluate on Your Traffic](https://docs.litellm.ai/docs/auto_router/evaluate.md): Run a shadow evaluation to judge the router against your current model on live traffic before switching, then read what it saved in the usage tab and per request.
- [Auto Router OTEL Telemetry](https://docs.litellm.ai/docs/auto_router/telemetry.md): Trace the Auto Router configuration, selected model, routing reason, and recovered classifier failures. Enable OpenTelemetry once on the gateway and inspect the results in Lens...
- [Auto Router Feature History](https://docs.litellm.ai/docs/auto_router/feature_history.md): Which Auto Router features shipped in which LiteLLM release, so you know what to expect when you upgrade.
- [[Beta] Auto Routing](https://docs.litellm.ai/docs/proxy/auto_routing.md): One router for complexity, semantic, and adaptive routing. Classify each request with heuristics, an LLM classifier, JEV through TypeSafe System One Choice, lexical/semantic...
- [Semantic Auto Router (deprecated)](https://docs.litellm.ai/docs/proxy/auto_routing_semantic.md): The semantic Auto Router is superseded by Auto Routing, which folds semantic keyword matching, complexity scoring, and adaptive routing into a single...
- [Auto Routing Benchmark: Cost Ladders](https://docs.litellm.ai/docs/proxy/auto_routing_benchmark.md): This benchmark evaluates how well LiteLLM routing strategies reduce cost while preserving response quality. It compares the semantic Auto Router, the rule-based...
- [[BETA] Adaptive Router](https://docs.litellm.ai/docs/adaptive_router.md): Beta feature. Share feedback on Discord or Slack.

## Python SDK

- [LiteLLM Python SDK](https://docs.litellm.ai/docs/python_sdk.md): The LiteLLM Python SDK gives you one interface to call 100+ LLM providers from Python, with the same input and output format for each provider.
- [SDK Quickstart](https://docs.litellm.ai/docs/learn/sdk_quickstart.md): Make your first LiteLLM SDK call, then jump to the right docs for the next feature you need.
- [LiteLLM Core](https://docs.litellm.ai/docs/litellm_core.md): Install the LiteLLM SDK with fewer default dependencies and add AWS and Python tokenizer packages when needed
- [Input Params](https://docs.litellm.ai/docs/completion/input.md)
- [/embeddings](https://docs.litellm.ai/docs/embedding/supported_embedding.md)
- [/responses](https://docs.litellm.ai/docs/response_api.md): LiteLLM provides an endpoint in the spec of OpenAI's /responses API
- [/completions](https://docs.litellm.ai/docs/text_completion.md)
- [Image Generations](https://docs.litellm.ai/docs/image_generation.md)
- [Prompt Compression (compress())](https://docs.litellm.ai/docs/completion/prompt_compression.md): This feature is in beta. APIs and behavior may change before general availability.
- [/audio/transcriptions](https://docs.litellm.ai/docs/audio_transcription.md)
- [/audio/speech](https://docs.litellm.ai/docs/text_to_speech.md)
- [Sandbox / Code Execution](https://docs.litellm.ai/docs/sandbox.md): Run model-generated code inside an isolated sandbox and get its output back. The API is provider-agnostic; e2b and opensandbox are supported backends, talked to directly over...
- [Agents (litellm.agent)](https://docs.litellm.ai/docs/harness.md): Run Claude Code, Codex, OpenCode, Deep Agents and Tool Loop from Python with litellm.agent(), with model calls routed through the LiteLLM AI Gateway.
- [Using with LiteLLM AI Gateway](https://docs.litellm.ai/docs/harness/gateway.md): Run agents with litellm.agent() against one LiteLLM AI Gateway virtual key, pick the right harness for each job, and track spend per harness.
- [Supported harnesses](https://docs.litellm.ai/docs/harness/supported.md): This release supports five harnesses. Each one is driven through its own programmatic interface, never by scraping a terminal
- [Claude Code](https://docs.litellm.ai/docs/harness/claude_code.md): Harness.CLAUDE_CODE runs Anthropic's Claude Code CLI with claude -p --output-format stream-json inside your sandbox and turns its stream-json output into events.
- [Codex](https://docs.litellm.ai/docs/harness/codex.md): Harness.CODEX runs OpenAI's Codex CLI with codex exec --json inside your sandbox and turns its JSONL events into events.
- [OpenCode](https://docs.litellm.ai/docs/harness/opencode.md): Harness.OPENCODE runs the OpenCode CLI with opencode run --format json inside your sandbox and turns its JSON events into events.
- [Deep Agents](https://docs.litellm.ai/docs/harness/deepagents.md): Harness.DEEPAGENTS runs LangChain's Deep Agents in your Python process. There's no process to launch and nothing to install in the sandbox
- [Tool Loop](https://docs.litellm.ai/docs/harness/tool_loop.md): Harness.TOOL_LOOP runs a minimal tool-calling loop in your Python process. It calls litellm.acompletion(), validates tool arguments against your Python function signatures, runs...
- [Quickstart](https://docs.litellm.ai/docs/harness/quickstart.md)
- [Models and routing](https://docs.litellm.ai/docs/harness/models.md): model= follows the same convention as litellm.completion. A model prefixed with litellm_proxy/ goes to the gateway, and anything else is called directly through the LiteLLM SDK.
- [Sessions](https://docs.litellm.ai/docs/harness/sessions.md): A session keeps its working directory and conversation history alive between turns. litellm.agent() creates a session for one turn and closes it afterwards. Use...
- [Events](https://docs.litellm.ai/docs/harness/events.md): litellm.agent(..., stream=True) yields eight kinds of event. Each adapter translates its runtime's native events into these.
- [Sandboxes](https://docs.litellm.ai/docs/harness/sandboxes.md): A sandbox is required by every call. CLI harnesses run there, and Deep Agents' built-in file and shell tools use it. Tool Loop and Deep Agents run Python tools in your process,...
- [Tools and skills](https://docs.litellm.ai/docs/harness/tools.md)
- [Permissions](https://docs.litellm.ai/docs/harness/permissions.md): permissions= takes a mode, and each harness sets up its permission behavior to match. The default is "full". The sandbox only limits operations that run inside it; Python tools...
- [Structured output](https://docs.litellm.ai/docs/harness/structured_output.md): Pass a Pydantic model as output=. The runtime does its normal work, and its final answer is validated into result.output.
- [API reference](https://docs.litellm.ai/docs/harness/api_reference.md): The entry points live on the top-level litellm module, next to litellm.completion. Each one has an async version with an a prefix.
- [Errors](https://docs.litellm.ai/docs/harness/errors.md): Setup problems raise an exception. Limits you set, like timeout and maxturns, end the turn and return a Result with a stopreason, so partial work is kept.
- [Setting API Keys, Base, Version](https://docs.litellm.ai/docs/set_keys.md): LiteLLM allows you to specify the following:
- [SDK Proxy Authentication (OAuth2/JWT Auto-Refresh)](https://docs.litellm.ai/docs/proxy_auth.md): Automatically obtain and refresh OAuth2/JWT tokens when using the LiteLLM Python SDK with a LiteLLM Proxy that requires JWT authentication.
- [Caching - In-Memory, Redis, s3, gcs, Redis Semantic Cache, Disk](https://docs.litellm.ai/docs/caching/all_caches.md)
- [Completion Token Usage & Cost](https://docs.litellm.ai/docs/completion/token_usage.md): By default LiteLLM returns token usage in all completion requests (See here)
- [Trimming Input Messages](https://docs.litellm.ai/docs/completion/message_trimming.md): Use litellm.trim_messages() to ensure messages does not exceed a model's token limit or specified max_tokens
- [Model Alias](https://docs.litellm.ai/docs/completion/model_alias.md): The model name you show an end-user might be different from the one you pass to LiteLLM - e.g. Displaying GPT-5.6 while calling {} on the backend.
- [Prompt Formatting](https://docs.litellm.ai/docs/completion/prompt_formatting.md): LiteLLM automatically translates the OpenAI ChatCompletions prompt format, to other models. You can control this by setting a custom prompt template for a model as well.
- [Reliability - Retries, Fallbacks](https://docs.litellm.ai/docs/completion/reliable_completions.md): LiteLLM helps prevent failed requests in 2 ways:
- [Mock Completion() Responses - Save Testing Costs](https://docs.litellm.ai/docs/completion/mock_requests.md): For testing purposes, you can use completion() with mock_response to mock calling the completion endpoint.
- [Exception Mapping](https://docs.litellm.ai/docs/exception_mapping.md): LiteLLM maps exceptions across all providers to their OpenAI counterparts.

## LiteLLM Lens

- [LiteLLM Lens](https://docs.litellm.ai/docs/proxy/lens.md): Trace agent runs and investigate recurring failures with LiteLLM Lens.
- [Set up Lens](https://docs.litellm.ai/docs/proxy/lens/deployment.md): Use Lens in your LiteLLM dashboard or deploy it as a separate application.
- [Start standalone Lens](https://docs.litellm.ai/docs/proxy/lens/deployment/local.md): Run Lens and ClickHouse locally, sign in, and send a first trace without a LiteLLM gateway.
- [Add Lens to LiteLLM](https://docs.litellm.ai/docs/proxy/lens/deployment/litellm.md): Connect a running Lens service to your gateway and open it from the LiteLLM sidebar.
- [Kubernetes](https://docs.litellm.ai/docs/proxy/lens/deployment/kubernetes.md): Deploy independent Lens with its own Helm chart and optional LiteLLM connection.
- [Docker Compose on a server](https://docs.litellm.ai/docs/proxy/lens/deployment/server.md): Run standalone Lens and ClickHouse on one server with persistent storage and HTTPS.
- [Connect Lens to Docker Compose](https://docs.litellm.ai/docs/proxy/lens/deployment/docker-compose.md): Add a running Lens service to your existing LiteLLM Compose deployment.
- [Lens containers](https://docs.litellm.ai/docs/proxy/lens/deployment/docker.md): Run the independent Lens container with ClickHouse and optionally connect LiteLLM.
- [Storage and secrets](https://docs.litellm.ai/docs/proxy/lens/deployment/storage.md): Configure supported ClickHouse storage, Lens credentials and GitOps.
- [Configuration and troubleshooting](https://docs.litellm.ai/docs/proxy/lens/deployment/configuration.md): Configure standalone Lens credentials, networking and the optional LiteLLM connection.
- [Releases and images](https://docs.litellm.ai/docs/proxy/lens/deployment/releases.md): Understand paired Lens and LiteLLM releases, standalone deployment, and source builds.
- [Upgrade Lens](https://docs.litellm.ai/docs/proxy/lens/deployment/upgrades.md): Update Lens independently while retaining its records, credentials and recovery point.
- [Build from source](https://docs.litellm.ai/docs/proxy/lens/deployment/development.md): Develop standalone Lens with its Rust backend, ClickHouse and shared UI.
- [Send your first trace](https://docs.litellm.ai/docs/proxy/lens/first-trace.md): Send one model call to Lens, then connect your existing agent. Works with standalone Lens and LiteLLM.
- [Framework examples through LiteLLM](https://docs.litellm.ai/docs/proxy/lens/framework-examples.md): Complete framework examples for a model connection through LiteLLM.
- [DeepAgents](https://docs.litellm.ai/docs/proxy/lens/integrations/deepagents.md): Run DeepAgents examples and send their agent traces to LiteLLM Lens.
- [LangGraph](https://docs.litellm.ai/docs/proxy/lens/integrations/langgraph.md): Run LangGraph examples and send their agent traces to LiteLLM Lens.
- [LangChain](https://docs.litellm.ai/docs/proxy/lens/integrations/langchain.md): Run LangChain examples and send their agent traces to LiteLLM Lens.
- [OpenAI Agents SDK](https://docs.litellm.ai/docs/proxy/lens/integrations/openai-agents.md): Run OpenAI Agents SDK examples and send their agent traces to LiteLLM Lens.
- [Claude Agent SDK](https://docs.litellm.ai/docs/proxy/lens/integrations/claude-agent-sdk.md): Run Claude Agent SDK examples and send their agent traces to LiteLLM Lens.
- [CrewAI](https://docs.litellm.ai/docs/proxy/lens/integrations/crewai.md): Run CrewAI examples and send their agent traces to LiteLLM Lens.
- [Pydantic AI](https://docs.litellm.ai/docs/proxy/lens/integrations/pydantic-ai.md): Run Pydantic AI examples and send their agent traces to LiteLLM Lens.
- [LlamaIndex](https://docs.litellm.ai/docs/proxy/lens/integrations/llamaindex.md): Run LlamaIndex examples and send their agent traces to LiteLLM Lens.
- [Google ADK](https://docs.litellm.ai/docs/proxy/lens/integrations/google-adk.md): Run Google ADK examples and send their agent traces to LiteLLM Lens.
- [Strands Agents](https://docs.litellm.ai/docs/proxy/lens/integrations/strands.md): Run Strands Agents examples and send their agent traces to LiteLLM Lens.
- [Vercel AI SDK (TypeScript)](https://docs.litellm.ai/docs/proxy/lens/integrations/vercel-ai-sdk-js.md): Run Vercel AI SDK (TypeScript) examples and send their agent traces to LiteLLM Lens.
- [Vercel AI SDK (Python)](https://docs.litellm.ai/docs/proxy/lens/integrations/vercel-ai-sdk-py.md): Run Vercel AI SDK (Python) examples and send their agent traces to LiteLLM Lens.
- [Mastra](https://docs.litellm.ai/docs/proxy/lens/integrations/mastra.md): Run Mastra examples and send their agent traces to LiteLLM Lens.
- [Hermes Agent](https://docs.litellm.ai/docs/proxy/lens/integrations/hermes-agent.md): Run Hermes Agent examples and send their agent traces to LiteLLM Lens.
- [OpenTelemetry](https://docs.litellm.ai/docs/proxy/lens/integrations/opentelemetry.md): Run OpenTelemetry examples and send their agent traces to LiteLLM Lens.
- [OpenClaw](https://docs.litellm.ai/docs/proxy/lens/integrations/openclaw.md): Send OpenClaw agent activity to LiteLLM Lens.
- [Hermes](https://docs.litellm.ai/docs/proxy/lens/integrations/hermes.md): Send Hermes agent activity to LiteLLM Lens.
- [Trace coding agent sessions](https://docs.litellm.ai/docs/proxy/lens/coding-agents.md): Send your personal Claude Code and Codex sessions to LiteLLM Lens.
- [Claude Code](https://docs.litellm.ai/docs/proxy/lens/coding-agents/claude-code.md): Connect personal Claude Code sessions to LiteLLM Lens using the maintained setup guides.
- [Codex](https://docs.litellm.ai/docs/proxy/lens/coding-agents/codex.md): Connect personal Codex sessions to LiteLLM Lens using the maintained setup guides.
- [Investigations](https://docs.litellm.ai/docs/proxy/lens/investigations.md): Review agent activity, investigate failures, and read the supporting evidence.
- [API reference](https://docs.litellm.ai/docs/proxy/lens/api.md): Ingest and read agent traces, start investigations, and retrieve Lens findings.
- [Historical scalability measurements](https://docs.litellm.ai/docs/proxy/lens/scalability.md): Archived measurements of gateway-hosted Lens, with the current standalone benchmark procedure.

## Features

- [Features](https://docs.litellm.ai/docs/guides/index.md): Features are what LiteLLM does with your requests across providers: load balancing and fallbacks, streaming, tool calling, media input, prompt caching, and more. Most pages show...

## Routing & Reliability

- [Proxy - Load Balancing](https://docs.litellm.ai/docs/proxy/load_balancing.md): Load balance multiple instances of the same model
- [Router - Load Balancing](https://docs.litellm.ai/docs/routing.md): LiteLLM Router for load balancing, routing, retries, cooldowns, and fallbacks (failover) across multiple LLM deployments and providers.
- [Provider specific Wildcard routing](https://docs.litellm.ai/docs/wildcard_routing.md): Proxy all models from a provider
- [Health Check Driven Routing](https://docs.litellm.ai/docs/proxy/health_check_routing.md): Route traffic away from unhealthy deployments before users hit errors. Background health checks run on a configurable interval, and any deployment that fails gets removed from...
- [[BETA] Request Prioritization](https://docs.litellm.ai/docs/scheduler.md): Beta feature. Use for testing only.
- [Fallbacks (Provider Failover)](https://docs.litellm.ai/docs/proxy/reliability.md): Set up automatic provider failover in LiteLLM. If a model or provider fails after num_retries, fallback to another model group for high availability and reliability.
- [Client-Side Fallbacks](https://docs.litellm.ai/docs/proxy/client_side_fallbacks.md): Set fallback models per request from the client or the Python SDK, and control the prompt sent to each fallback model.
- [Timeouts](https://docs.litellm.ai/docs/proxy/timeout.md): The timeout set in router is for the entire length of the call, and is passed down to the completion() call level as well.
- [[New] Fallback Management Endpoints](https://docs.litellm.ai/docs/proxy/fallback_management.md): Dedicated endpoints for managing model fallbacks separately from the general configuration.
- [Tag Based Routing](https://docs.litellm.ai/docs/proxy/tag_routing.md)
- [Budget Routing](https://docs.litellm.ai/docs/proxy/provider_budget_routing.md): LiteLLM Supports setting the following budgets:
- [UI - Router Settings for Keys and Teams](https://docs.litellm.ai/docs/proxy/keys_teams_router_settings.md): Configure router settings at the key and team level to achieve granular control over routing behavior, fallbacks, retries, and other router configurations. This enables you to...
- [Manage Routing Groups](https://docs.litellm.ai/docs/proxy/ui/routing_groups.md): Routing groups let you apply different routing strategies to different models in the same router, for example latency-based routing for {} while cheaper models use...
- [Routing Plugins](https://docs.litellm.ai/docs/routing_plugins.md): Routing plugins are available from v1.92.x (Thursday's release). The design is still evolving, so tell us how you'd use it and what you'd want next in the autorouter discussion...
- [Fusion model](https://docs.litellm.ai/docs/fusion.md): Use litellm/fusion-1 for judge-synthesized answers from a parallel model panel.

## Requests & Responses

- [Streaming + Async](https://docs.litellm.ai/docs/completion/stream.md): | Feature | LiteLLM SDK | LiteLLM Proxy |
- [Batching Completion()](https://docs.litellm.ai/docs/completion/batching.md)
- [Structured Outputs (JSON Mode)](https://docs.litellm.ai/docs/completion/json_mode.md)
- ['Thinking' / 'Reasoning Content'](https://docs.litellm.ai/docs/reasoning_content.md)

## Tool Calling

- [Function Calling](https://docs.litellm.ai/docs/completion/function_call.md): Checking if a model supports function calling
- [Web Search](https://docs.litellm.ai/docs/completion/web_search.md): Use web search with litellm
- [Web Search Integration](https://docs.litellm.ai/docs/integrations/websearch_interception.md): Enable transparent server-side web search execution for any LLM provider. LiteLLM automatically intercepts web search tool calls and executes them using your configured search...
- [Web Fetch](https://docs.litellm.ai/docs/completion/web_fetch.md): The web fetch tool allows LLMs to retrieve full content from specified web pages and PDF documents. This enables AI models to access real-time information from the internet and...
- [Computer Use](https://docs.litellm.ai/docs/completion/computer_use.md): Computer use allows models to interact with computer interfaces by taking screenshots and performing actions like clicking, typing, and scrolling. This enables AI models to...
- [Code Interpreter](https://docs.litellm.ai/docs/guides/code_interpreter.md): Use OpenAI's Code Interpreter tool to execute Python code in a secure, sandboxed environment.
- [Advisor Tool](https://docs.litellm.ai/docs/completion/anthropic_advisor_tool.md): Pair a faster executor model with a higher-intelligence advisor model that provides strategic guidance mid-generation.
- [Using Vector Stores (Knowledge Bases)](https://docs.litellm.ai/docs/completion/knowledgebase.md)

## Multimodal

- [Using Vision Models](https://docs.litellm.ai/docs/completion/vision.md)
- [Using Audio Models](https://docs.litellm.ai/docs/completion/audio.md): How to send / receive audio to a /chat/completions endpoint
- [Using PDF Input](https://docs.litellm.ai/docs/completion/document_understanding.md): How to send / receive pdf's (other document types) to a /chat/completions endpoint
- [Image Generation in Chat Completions, Responses API](https://docs.litellm.ai/docs/completion/image_generation_chat.md): This guide covers how to generate images when using the chat/completions. Note - if you want this on Responses API please file a Feature Request here.

## Prompts & Context

- [Prompt Caching](https://docs.litellm.ai/docs/completion/prompt_caching.md)
- [Pre-fix Assistant Messages](https://docs.litellm.ai/docs/completion/prefix.md)
- [Predicted Outputs](https://docs.litellm.ai/docs/completion/predict_outputs.md): | Property | Details |

## Compatibility

- [Provider-specific Params](https://docs.litellm.ai/docs/completion/provider_specific_params.md): Providers might offer params not supported by OpenAI (e.g. top_k). LiteLLM treats any non-openai param, as a provider-specific param, and passes it to the provider in the request...
- [Drop Unsupported Params](https://docs.litellm.ai/docs/completion/drop_params.md): Drop unsupported OpenAI params by your LLM Provider.
- [Message Sanitization for Tool Calling for anthropic models](https://docs.litellm.ai/docs/completion/message_sanitization.md): Automatically fix common message formatting issues when using tool calling with modify_params=True

## liteagents SDK

- [liteagents SDK](https://docs.litellm.ai/docs/liteagents.md): Switch between DeepAgents, Pydantic AI, Claude Agent SDK, Codex, and OpenCode through one Python SDK, with the same application code and your choice of model.

## lite CLI

- [LiteLLM Proxy CLI](https://docs.litellm.ai/docs/proxy/management_cli.md): The lite CLI is a command-line tool for managing your LiteLLM proxy
- [lite autoroute](https://docs.litellm.ai/docs/learn/autorouter_cli.md): Try LiteLLM's complexity-based auto-routing locally against your real proxy with the lite CLI, and route Claude Code traffic through it.

## Self-Hosted Coding Agents

- [Moyai](https://docs.litellm.ai/docs/self_hosted_coding_agents/moyai.md): Run open source coding agents in your cloud, choose models through LiteLLM, and track each teammate's costs. See Moyai in action and set up your first task.
- [Set up Moyai](https://docs.litellm.ai/docs/self_hosted_coding_agents/moyai/setup.md): Deploy Moyai on Modal, connect your LiteLLM gateway, verify your first cloud task, and connect a GitHub repository.
- [Moyai architecture](https://docs.litellm.ai/docs/self_hosted_coding_agents/moyai/architecture.md): How Moyai separates orchestration, agent sandboxes, and model access, with checkpoints, worker recovery, and handling for uncertain tool outcomes.

## Supported Endpoints

- [/assistants](https://docs.litellm.ai/docs/assistants.md): OpenAI has deprecated the Assistants API. It will shut down on August 26, 2026.
- [/batches](https://docs.litellm.ai/docs/batches.md)
- [[BETA] LiteLLM Managed Files with Batches](https://docs.litellm.ai/docs/proxy/managed_batches.md): | Feature | Supported | Comments |
- [Unmanaged Vertex AI Batches](https://docs.litellm.ai/docs/proxy/unmanaged_vertex_batches.md): LiteLLM supports two paths for Vertex AI batch jobs. The managed path handles file upload and format conversion automatically. The unmanaged path lets you upload batch files...
- [/containers](https://docs.litellm.ai/docs/containers.md): Manage OpenAI and Azure OpenAI code interpreter containers (sessions) for executing code in isolated environments.
- [/containers/files](https://docs.litellm.ai/docs/container_files.md): Manage files within Code Interpreter containers. Files are created automatically when code interpreter generates outputs (charts, CSVs, images, etc.).
- [Output](https://docs.litellm.ai/docs/completion/output.md)
- [Usage](https://docs.litellm.ai/docs/completion/usage.md): LiteLLM returns the OpenAI compatible usage object across all providers.
- [Custom HTTP Handler](https://docs.litellm.ai/docs/completion/http_handler_config.md): Configure custom aiohttp sessions for better performance and control in LiteLLM completions.
- [/converse](https://docs.litellm.ai/docs/bedrock_converse.md): Call Bedrock's /converse endpoint through LiteLLM Proxy.
- [Provider Files Endpoints](https://docs.litellm.ai/docs/files_endpoints.md): Files are used to upload documents that can be used with features like Assistants, Fine-tuning, and Batch API.
- [[BETA] LiteLLM Managed Files](https://docs.litellm.ai/docs/proxy/litellm_managed_files.md): - Reuse the same file across different providers.
- [/fine_tuning](https://docs.litellm.ai/docs/fine_tuning.md): | Feature | Supported | Notes |
- [[BETA] LiteLLM Managed Files with Finetuning](https://docs.litellm.ai/docs/proxy/managed_finetuning.md): | Property | Value | Comments |
- [/evals](https://docs.litellm.ai/docs/evals_api.md): LiteLLM Proxy supports OpenAI's Evaluations (Evals) API, allowing you to create, manage, and run evaluations to measure model performance against defined testing criteria.
- [/generateContent](https://docs.litellm.ai/docs/generateContent.md): Use LiteLLM to call Google AI's generateContent endpoints for text generation, multimodal interactions, and streaming responses.
- [/guardrails/apply_guardrail](https://docs.litellm.ai/docs/apply_guardrail.md): Use this endpoint to directly call a guardrail configured on your LiteLLM instance. This is useful when you have services that need to directly call a guardrail.
- [/invoke](https://docs.litellm.ai/docs/bedrock_invoke.md): Call Bedrock's /invoke endpoint through LiteLLM Proxy.
- [/interactions](https://docs.litellm.ai/docs/interactions.md): | Feature | Supported | Notes |
- [Google AI Studio - Managed Agents](https://docs.litellm.ai/docs/managed_agents.md): LiteLLM now supports the Google AI Studio Managed Agents API. Create, manage, and run custom agents through LiteLLM.
- [/memory](https://docs.litellm.ai/docs/memory_management.md): CRUD endpoints for storing and retrieving user/team-scoped memory entries on the LiteLLM proxy. Use these to persist conversation context, agent memory, team playbooks, or any...
- [/images/edits](https://docs.litellm.ai/docs/image_edits.md): LiteLLM provides image editing functionality that maps to OpenAI's /images/edits API endpoint. Now supports both single and multiple image editing.
- [[BETA] Image Variations](https://docs.litellm.ai/docs/image_variations.md): OpenAI's /image/variations endpoint is now supported.
- [/videos](https://docs.litellm.ai/docs/videos.md): | Feature | Supported |
- [Veo Video Generation with Google AI Studio](https://docs.litellm.ai/docs/proxy/veo_video_generation.md): Generate videos using Google's Veo model through LiteLLM's pass-through endpoints.
- [Vector Stores - Overview](https://docs.litellm.ai/docs/vector_stores/index.md): LiteLLM has three distinct ways to work with vector stores. They are separate APIs with separate endpoints, and knowing which one you need saves a lot of confusion:
- [LiteLLM Managed Vector Stores](https://docs.litellm.ai/docs/vector_stores/managed_vector_stores.md): Register an existing provider vector store (Bedrock Knowledge Base, Vertex AI Search datastore, Azure AI Search index, Milvus collection, Valkey search index, MongoDB Vector...
- [/vector_stores - Create Vector Store](https://docs.litellm.ai/docs/vector_stores/create.md): Create a vector store which can be used to store and search document chunks for retrieval-augmented generation (RAG) use cases.
- [/vector_stores/search - Search Vector Store](https://docs.litellm.ai/docs/vector_stores/search.md): Search a vector store for relevant chunks based on a query and file attributes filter. This is useful for retrieval-augmented generation (RAG) use cases.
- [/vector_stores/\{vector_store_id\}/files](https://docs.litellm.ai/docs/vector_store_files.md): Vector store files represent the individual files that live inside a vector store.
- [/rag/ingest](https://docs.litellm.ai/docs/rag_ingest.md): All-in-one document ingestion pipeline: Upload → Chunk → Embed → Vector Store
- [/rag/query](https://docs.litellm.ai/docs/rag_query.md): RAG Query endpoint: Search Vector Store → (Rerank) → LLM Completion
- [/v1/messages](https://docs.litellm.ai/docs/anthropic_unified/index.md): Use LiteLLM to call all your LLM APIs in the Anthropic v1/messages format.
- [Structured Output /v1/messages](https://docs.litellm.ai/docs/anthropic_unified/structured_output.md): Use LiteLLM to call Anthropic's structured output feature via the /v1/messages endpoint.
- [v1/messages → /responses Parameter Mapping](https://docs.litellm.ai/docs/anthropic_unified/messages_to_responses_mapping.md): When you send a request to /v1/messages targeting an OpenAI or Azure model, LiteLLM internally routes it through the OpenAI Responses API. This page documents exactly how every...
- [Native /v1/messages and /v1/responses Passthrough for OpenAI-Compatible Providers](https://docs.litellm.ai/docs/anthropic_unified/native_passthrough.md): When a deployment's provider has no native Anthropic Messages support, LiteLLM translates each /v1/messages request into the provider's own API cachecontrol blocks are dropped,...
- [Token Counting](https://docs.litellm.ai/docs/count_tokens.md)
- [/v1/messages/count_tokens](https://docs.litellm.ai/docs/anthropic_count_tokens.md)
- [/v1/decisions and /v1/systemone](https://docs.litellm.ai/docs/decisions.md): Decision models answer typed questions about an input and return probabilities instead of text: a yes/no probability, a pick from a fixed list, a score on a scale, or on TypeSafe...
- [/moderations](https://docs.litellm.ai/docs/moderation.md)
- [/ocr](https://docs.litellm.ai/docs/ocr.md): | Feature | Supported |
- [Why Pass-Through Endpoints?](https://docs.litellm.ai/docs/pass_through/intro.md): These endpoints are useful for 2 scenarios:
- [Anthropic Passthrough](https://docs.litellm.ai/docs/pass_through/anthropic_completion.md): Pass-through endpoints for Anthropic - call provider-specific endpoint, in native format (no translation).
- [AssemblyAI](https://docs.litellm.ai/docs/pass_through/assembly_ai.md): Pass-through endpoints for AssemblyAI - call AssemblyAI endpoints, in native format (no translation).
- [Bedrock (boto3) SDK](https://docs.litellm.ai/docs/pass_through/bedrock.md): Pass-through endpoints for Bedrock - call provider-specific endpoint, in native format (no translation).
- [Azure Passthrough](https://docs.litellm.ai/docs/pass_through/azure_passthrough.md)
- [Azure AI Speech](https://docs.litellm.ai/docs/pass_through/azure_speech.md): Pass-through endpoints for Azure AI Speech speech to text: the short audio REST recognition endpoint and the batch transcription REST API, in native Azure format (no...
- [Cohere SDK](https://docs.litellm.ai/docs/pass_through/cohere.md): Pass-through endpoints for Cohere - call provider-specific endpoint, in native format (no translation).
- [Amazon Comprehend Medical](https://docs.litellm.ai/docs/pass_through/comprehend_medical.md): Pass-through endpoints for Amazon Comprehend Medical - detect entities, PHI, and medical ontology links in clinical text, in native AWS format (no translation).
- [Cursor Cloud Agents](https://docs.litellm.ai/docs/pass_through/cursor.md): Pass-through endpoints for the Cursor Cloud Agents API. Launch and manage cloud agents that work on your repositories, in native format (no translation).
- [Deepgram Realtime (/listen) WebSocket Passthrough](https://docs.litellm.ai/docs/pass_through/deepgram_listen_websocket.md): Stream live audio to Deepgram's realtime speech-to-text API (wss://api.deepgram.com/v1/listen) through the LiteLLM proxy. The proxy authenticates the caller with a LiteLLM...
- [GigaChat](https://docs.litellm.ai/docs/pass_through/gigachat.md): Pass-through endpoints for GigaChat - call the provider-specific endpoint in native format (no translation).
- [Google AI Studio SDK](https://docs.litellm.ai/docs/pass_through/google_ai_studio.md): Pass-through endpoints for Google AI Studio - call provider-specific endpoint, in native format (no translation).
- [Langfuse SDK](https://docs.litellm.ai/docs/pass_through/langfuse.md): Pass-through endpoints for Langfuse - call langfuse endpoints with LiteLLM Virtual Key.
- [Mistral](https://docs.litellm.ai/docs/pass_through/mistral.md): Pass-through endpoints for Mistral - call provider-specific endpoint, in native format (no translation).
- [OpenAI Passthrough](https://docs.litellm.ai/docs/pass_through/openai_passthrough.md): Pass-through endpoints for direct OpenAI API access
- [TinyFish Agent](https://docs.litellm.ai/docs/pass_through/tinyfish.md): Pass-through endpoints for the TinyFish Agent API, goal-based web automation in native format (no translation). Point the official TinyFish SDK at the proxy and it works unchanged
- [Amazon Transcribe](https://docs.litellm.ai/docs/pass_through/transcribe.md): Pass-through endpoints for the Amazon Transcribe batch and management API (start, poll and delete transcription jobs, manage custom vocabularies, vocabulary filters, language...
- [TypeSafe AI (Jev)](https://docs.litellm.ai/docs/pass_through/typesafe.md): Pass-through endpoint for the TypeSafe AI System One API. Jev returns typed decisions (a choice, a score, or a yes/no probability) instead of text, so it is called through its...
- [Passthrough Managed IDs](https://docs.litellm.ai/docs/proxy/passthrough_managed_ids.md): When you use LiteLLM's passthrough endpoints (e.g. /openaipassthrough/v1/files, /azure/openai/batches) the upstream provider returns its own raw IDs such as file-abc123 or...
- [Vertex AI SDK](https://docs.litellm.ai/docs/pass_through/vertex_ai.md): Pass-through endpoints for Vertex AI - call provider-specific endpoint, in native format (no translation).
- [Vertex AI Live API WebSocket Passthrough](https://docs.litellm.ai/docs/pass_through/vertex_ai_live_websocket.md): LiteLLM now supports WebSocket passthrough for the Vertex AI Live API, enabling real-time bidirectional communication with Gemini models.
- [Vertex AI Search Datastores (Pass-through)](https://docs.litellm.ai/docs/pass_through/vertex_ai_search_datastores.md): Call the Vertex AI Discovery Engine Search API through LiteLLM, using Google's native request and response shapes.
- [VLLM](https://docs.litellm.ai/docs/pass_through/vllm.md): Pass-through endpoints for VLLM - call provider-specific endpoint, in native format (no translation).
- [Create Pass Through Endpoints](https://docs.litellm.ai/docs/proxy/pass_through.md): Route requests from your LiteLLM proxy to any external API. Perfect for custom models, image generation APIs, or any service you want to proxy through LiteLLM.
- [Pass-Through Cost & Usage Tracking](https://docs.litellm.ai/docs/proxy/pass_through_cost_tracking.md): Some pass-through targets fan a single HTTP request out to several models internally. LiteLLM cannot price those requests from the response body, so before this contract existed...
- [Guardrails on Pass-Through Endpoints](https://docs.litellm.ai/docs/proxy/pass_through_guardrails.md)
- [/realtime](https://docs.litellm.ai/docs/realtime.md): Use this to loadbalance across Azure + OpenAI + xAI and more.
- [/realtime - WebRTC Support](https://docs.litellm.ai/docs/proxy/realtime_webrtc.md): Connect to the Realtime API via WebRTC from browser/mobile clients. LiteLLM handles auth; audio streams directly to OpenAI/Azure.
- [/rerank](https://docs.litellm.ai/docs/rerank.md): LiteLLM Follows the cohere api request / response for the rerank api
- [Prompt Management with Responses API](https://docs.litellm.ai/docs/prompt_management.md): Use LiteLLM Prompt Management with /v1/responses by passing promptid and optional promptvariables.
- [/responses/compact](https://docs.litellm.ai/docs/response_api_compact.md): Compress conversation history using OpenAI's /responses/compact endpoint.
- [Overview](https://docs.litellm.ai/docs/search/index.md): | Feature | Supported |
- [Perplexity AI Search](https://docs.litellm.ai/docs/search/perplexity.md)
- [Tavily Search](https://docs.litellm.ai/docs/search/tavily.md)
- [Exa AI Search](https://docs.litellm.ai/docs/search/exa_ai.md)
- [Brave Search](https://docs.litellm.ai/docs/search/brave.md): Get started by creating a free API key via https://brave.com/search/api/.
- [Parallel AI Search](https://docs.litellm.ai/docs/search/parallel_ai.md)
- [Google Programmable Search Engine (PSE)](https://docs.litellm.ai/docs/search/google_pse.md): Get API Key: Google Cloud Console
- [DataForSEO Search](https://docs.litellm.ai/docs/search/dataforseo.md)
- [Firecrawl Search](https://docs.litellm.ai/docs/search/firecrawl.md)
- [SearXNG Search](https://docs.litellm.ai/docs/search/searxng.md)
- [Linkup Search](https://docs.litellm.ai/docs/search/linkup.md)
- [Serper Search](https://docs.litellm.ai/docs/search/serper.md)
- [You.com Search](https://docs.litellm.ai/docs/search/you_com.md): Get API Key (optional, for higher rate limits)//you.com/docs
- [APISerpent Search](https://docs.litellm.ai/docs/search/apiserpent.md)
- [Amazon Bedrock AgentCore Web Search](https://docs.litellm.ai/docs/search/agentcore.md): Web Search on Amazon Bedrock AgentCore is an AWS-managed web index exposed as an MCP tool on an AgentCore Gateway, so search traffic stays inside AWS and is billed by AWS instead...
- [Nimble Search](https://docs.litellm.ai/docs/search/nimble.md)
- [Grounding with Bing Search (Microsoft Foundry)](https://docs.litellm.ai/docs/search/bing_grounding.md): Grounding with Bing Search runs a web search through a model deployment in a Microsoft Foundry project, using the project's Responses API. The model performs the search and...
- [/skills - Anthropic Skills API](https://docs.litellm.ai/docs/skills.md): | Feature | Supported |

## Providers

- [Calling Finetuned Models](https://docs.litellm.ai/docs/guides/finetuned_models.md)
- [OpenAI](https://docs.litellm.ai/docs/providers/openai.md): LiteLLM supports OpenAI Chat + Embedding calls.
- [OpenAI - Response API](https://docs.litellm.ai/docs/providers/openai/responses_api.md): To use Fast mode on supported OpenAI models or Ultrafast on GPT-6 Astra, see Fast & Ultrafast mode for Admin UI setup, configuration, and request examples.
- [OpenAI - Text-to-speech](https://docs.litellm.ai/docs/providers/openai/text_to_speech.md)
- [OpenAI Video Generation](https://docs.litellm.ai/docs/providers/openai/videos.md): LiteLLM supports OpenAI's video generation models including Sora.
- [OpenAI Fast & Ultrafast mode](https://docs.litellm.ai/docs/providers/openai/ultrafast.md): Set up OpenAI Fast and Ultrafast modes through the LiteLLM Admin UI or config.yaml, call them through the Responses API, and expose them in Codex.
- [OpenAI (Text Completion)](https://docs.litellm.ai/docs/providers/text_completion_openai.md): LiteLLM supports OpenAI text completion models
- [Anthropic](https://docs.litellm.ai/docs/providers/anthropic.md): LiteLLM supports all anthropic models.
- [Tool Search](https://docs.litellm.ai/docs/providers/anthropic_tool_search.md): Tool search enables Claude to dynamically discover and load tools on-demand from large tool catalogs (10,000+ tools). Instead of loading all tool definitions into the context...
- [Preserved Thinking Prefix Stability](https://docs.litellm.ai/docs/providers/anthropic_preserved_thinking.md): Claude models that preserve thinking across turns sign each thinking block and bind it to the request that produced it. When you replay the conversation with the...
- [Gemini - Google AI Studio](https://docs.litellm.ai/docs/providers/gemini.md): | Property | Details |
- [Gemini Video Generation (Veo)](https://docs.litellm.ai/docs/providers/gemini/videos.md): LiteLLM supports Google's Veo video generation models through a unified API interface.
- [Gemini Lyria (music generation)](https://docs.litellm.ai/docs/providers/gemini/music.md): Google Lyria 3 preview models are listed in LiteLLM’s model cost map under the gemini/ provider for metadata and spend tracking.
- [Gemini File Search](https://docs.litellm.ai/docs/providers/gemini_file_search.md): Use Google Gemini's File Search for Retrieval Augmented Generation (RAG) with LiteLLM.
- [[BETA] Google AI Studio (Gemini) Files API](https://docs.litellm.ai/docs/providers/google_ai_studio/files.md): Use this to upload files to Google AI Studio (Gemini).
- [Google AI Studio Image Generation](https://docs.litellm.ai/docs/providers/google_ai_studio/image_gen.md): Google AI Studio provides powerful image generation capabilities using Google's Imagen models to create high-quality images from text descriptions.
- [Gemini Realtime API - Google AI Studio](https://docs.litellm.ai/docs/providers/google_ai_studio/realtime.md): | Feature | Description | Comments |
- [VertexAI [Gemini]](https://docs.litellm.ai/docs/providers/vertex.md)
- [Vertex AI Video Generation (Veo)](https://docs.litellm.ai/docs/providers/vertex_ai/videos.md): LiteLLM supports Vertex AI's Veo video generation models using the unified OpenAI video API surface.
- [Vertex AI - Anthropic, DeepSeek, Model Garden, xAI](https://docs.litellm.ai/docs/providers/vertex_partner.md)
- [Vertex AI - Self Deployed Models](https://docs.litellm.ai/docs/providers/vertex_self_deployed.md): Deploy and use your own models on Vertex AI through Model Garden or custom endpoints.
- [Vertex AI Embedding](https://docs.litellm.ai/docs/providers/vertex_embedding.md)
- [Vertex AI Image Generation](https://docs.litellm.ai/docs/providers/vertex_image.md): Vertex AI supports two types of image generation:
- [Vertex AI Text to Speech](https://docs.litellm.ai/docs/providers/vertex_speech.md): | Property | Details |
- [Vertex AI Audio Transcription](https://docs.litellm.ai/docs/providers/vertex_transcription.md): | Property | Details |
- [Vertex Batch APIs](https://docs.litellm.ai/docs/providers/vertex_batch.md)
- [Vertex AI OCR](https://docs.litellm.ai/docs/providers/vertex_ocr.md)
- [Vertex AI Agent Engine](https://docs.litellm.ai/docs/providers/vertex_ai_agent_engine.md): Call Vertex AI Agent Engine (Reasoning Engines) in the OpenAI Request/Response format.
- [Vertex AI Gemini Live - Realtime API](https://docs.litellm.ai/docs/providers/vertex_realtime.md): Use Vertex AI's Gemini Live API (BidiGenerateContent) through LiteLLM's unified /realtime endpoint, which speaks the OpenAI Realtime protocol.
- [Azure OpenAI](https://docs.litellm.ai/docs/providers/azure/index.md)
- [Azure Responses API](https://docs.litellm.ai/docs/providers/azure/azure_responses.md): | Property | Details |
- [Azure OpenAI Embeddings](https://docs.litellm.ai/docs/providers/azure/azure_embedding.md)
- [Azure Text to Speech (tts)](https://docs.litellm.ai/docs/providers/azure/azure_speech.md)
- [Azure Video Generation](https://docs.litellm.ai/docs/providers/azure/videos.md): LiteLLM supports Azure OpenAI's video generation models including Sora with full end-to-end integration.
- [Azure AI Studio](https://docs.litellm.ai/docs/providers/azure_ai.md): LiteLLM supports all models on Azure AI Studio
- [Azure Model Router](https://docs.litellm.ai/docs/providers/azure_ai/azure_model_router.md): Azure Model Router is a feature in Azure AI Foundry that automatically routes your requests to the best available model based on your requirements. This allows you to use a...
- [Azure AI Foundry Agents](https://docs.litellm.ai/docs/providers/azure_ai_agents.md): Call Azure AI Foundry Agents in the OpenAI Request/Response format.
- [Azure AI OCR (Mistral, Cohere Parse)](https://docs.litellm.ai/docs/providers/azure_ocr.md)
- [Azure Document Intelligence OCR](https://docs.litellm.ai/docs/providers/azure_document_intelligence.md)
- [Azure AI Speech (Cognitive Services)](https://docs.litellm.ai/docs/providers/azure_ai_speech.md): Azure AI Speech is Azure's Cognitive Services text-to-speech API, separate from Azure OpenAI. It provides high-quality neural voices with broader language support and advanced...
- [Azure AI Image Generation (Black Forest Labs - Flux)](https://docs.litellm.ai/docs/providers/azure_ai_img.md): Azure AI provides powerful image generation capabilities using FLUX models from Black Forest Labs to create high-quality images from text descriptions.
- [Azure AI Search - Vector Store (Unified API)](https://docs.litellm.ai/docs/providers/azure_ai_vector_stores.md): Use this to search Azure AI Search Vector Stores, with LiteLLM's unified /chat/completions API.
- [Azure AI Search - Vector Store (Passthrough API)](https://docs.litellm.ai/docs/providers/azure_ai/azure_ai_vector_stores_passthrough.md): Use this to allow developers to create and search vector stores using the Azure AI Search API in the native Azure AI Search API format, without giving them the Azure AI...
- [AWS Bedrock](https://docs.litellm.ai/docs/providers/bedrock.md): ALL Bedrock models (Anthropic, Meta, Deepseek, Mistral, Amazon, etc.) are Supported
- [Bedrock Embedding](https://docs.litellm.ai/docs/providers/bedrock_embedding.md)
- [Bedrock Imported Models](https://docs.litellm.ai/docs/providers/bedrock_imported.md): Bedrock Imported Models (Deepseek, Deepseek R1, Qwen, OpenAI-compatible models)
- [AWS Bedrock - Image Generation](https://docs.litellm.ai/docs/providers/bedrock_image_gen.md): Use Bedrock for image generation with Stable Diffusion, Amazon Titan Image Generator, and Amazon Nova Canvas models.
- [AWS Bedrock - Rerank API](https://docs.litellm.ai/docs/providers/bedrock_rerank.md): Use Bedrock's Rerank API in the Cohere /rerank format.
- [Bedrock AgentCore](https://docs.litellm.ai/docs/providers/bedrock_agentcore.md): Call Bedrock AgentCore in the OpenAI Request/Response format.
- [Bedrock Agents](https://docs.litellm.ai/docs/providers/bedrock_agents.md): Call Bedrock Agents in the OpenAI Request/Response format.
- [Bedrock - Writer Palmyra](https://docs.litellm.ai/docs/providers/bedrock_writer.md)
- [Bedrock Batches](https://docs.litellm.ai/docs/providers/bedrock_batches.md): Use Amazon Bedrock Batch Inference API through LiteLLM.
- [Bedrock Realtime API](https://docs.litellm.ai/docs/providers/bedrock_realtime_with_audio.md)
- [AWS Polly Text to Speech (tts)](https://docs.litellm.ai/docs/providers/aws_polly.md)
- [Bedrock Knowledge Bases](https://docs.litellm.ai/docs/providers/bedrock_vector_store.md): AWS Bedrock Knowledge Bases allows you to connect your LLM's to your organization's data, letting your models retrieve and reference information specific to your business.
- [Amazon Bedrock Mantle](https://docs.litellm.ai/docs/providers/bedrock_mantle.md): Amazon Bedrock Mantle is Amazon Bedrock's distributed inference engine (Project Mantle) that exposes an OpenAI-compatible API for Bedrock-hosted models.
- [Mistral AI API](https://docs.litellm.ai/docs/providers/mistral.md)
- [Mistral AI Batch API](https://docs.litellm.ai/docs/providers/mistral_batches.md): LiteLLM routes the OpenAI-compatible /v1/files and /v1/batches endpoints to Mistral's Files and Batch APIs. A Mistral batch job runs one model for every line of the input file,...
- [xAI](https://docs.litellm.ai/docs/providers/xai.md)
- [xAI Voice Agent (Realtime API)](https://docs.litellm.ai/docs/providers/xai_realtime.md): xAI's Grok Voice Agent provides real-time voice conversation capabilities through WebSocket connections, enabling natural bidirectional audio interactions.
- [xAI Batch API](https://docs.litellm.ai/docs/providers/xai_batches.md): LiteLLM routes the OpenAI-compatible /v1/files and /v1/batches endpoints to xAI's Files and Batch APIs. xAI bills batch requests at a reduced rate, they do not count towards rate...
- [Deepseek](https://docs.litellm.ai/docs/providers/deepseek.md)
- [Groq](https://docs.litellm.ai/docs/providers/groq.md)
- [Ollama](https://docs.litellm.ai/docs/providers/ollama.md): LiteLLM supports all models from Ollama
- [VLLM](https://docs.litellm.ai/docs/providers/vllm.md): LiteLLM supports all models on VLLM.
- [vLLM - Batch + Files API](https://docs.litellm.ai/docs/providers/vllm_batches.md): vLLM's OpenAI-compatible server serves /v1/chat/completions, /v1/completions, /v1/embeddings, and /v1/responses, but it has no /v1/files or /v1/batches routes. LiteLLM fills that...
- [LM Studio](https://docs.litellm.ai/docs/providers/lm_studio.md)
- [Llamafile](https://docs.litellm.ai/docs/providers/llamafile.md): LiteLLM supports all models on Llamafile.
- [Docker Model Runner](https://docs.litellm.ai/docs/providers/docker_model_runner.md)
- [Lemonade](https://docs.litellm.ai/docs/providers/lemonade.md): Lemonade Server is an OpenAI-compatible local language model inference provider optimized for AMD GPUs and NPUs. The lemonade litellm provider supports standard chat completions...
- [Triton Inference Server](https://docs.litellm.ai/docs/providers/triton-inference-server.md): LiteLLM supports Embedding Models on Triton Inference Servers
- [Xinference [Xorbits Inference]](https://docs.litellm.ai/docs/providers/xinference.md)
- [Infinity](https://docs.litellm.ai/docs/providers/infinity.md): | Property | Details |
- [Nvidia NIM](https://docs.litellm.ai/docs/providers/nvidia_nim.md)
- [Nvidia NIM - Rerank](https://docs.litellm.ai/docs/providers/nvidia_nim_rerank.md): Use Nvidia NIM Rerank models through LiteLLM.
- [Petals](https://docs.litellm.ai/docs/providers/petals.md)
- [Custom API Server (Custom Format)](https://docs.litellm.ai/docs/providers/custom_llm_server.md): Call your custom torch-serve / internal LLM APIs via LiteLLM
- [OpenAI-Compatible Endpoints](https://docs.litellm.ai/docs/providers/openai_compatible.md): Selecting openai as the provider routes your request to an OpenAI-compatible endpoint using the upstream
- [LiteLLM Proxy (LLM Gateway)](https://docs.litellm.ai/docs/providers/litellm_proxy.md): | Property | Details |
- [OpenRouter](https://docs.litellm.ai/docs/providers/openrouter.md): LiteLLM supports all the text / chat / vision / embedding models from OpenRouter
- [Vercel AI Gateway](https://docs.litellm.ai/docs/providers/vercel_ai_gateway.md)
- [Helicone](https://docs.litellm.ai/docs/providers/helicone.md)
- [CometAPI](https://docs.litellm.ai/docs/providers/cometapi.md): LiteLLM supports all AI models from CometAPI. CometAPI provides access to 500+ AI models through a unified API interface, including cutting-edge models like GPT-5, Claude Opus...
- [CLF AI Gateway](https://docs.litellm.ai/docs/providers/clf_ai_gateway.md)
- [Poe](https://docs.litellm.ai/docs/providers/poe.md)
- [Milvus - Vector Store](https://docs.litellm.ai/docs/providers/milvus_vector_stores.md): Use Milvus as a vector store for RAG.
- [MongoDB - Vector Store (BETA)](https://docs.litellm.ai/docs/providers/mongodb_vector_stores.md): The MongoDB vector store integration is a BETA feature. It supports searching existing MongoDB Vector Search indexes through the optional LiteLLM MongoDB sidecar. Prepare your...
- [AWS S3 Vectors](https://docs.litellm.ai/docs/providers/s3_vectors.md): Use Amazon S3 Vectors as the vector store behind LiteLLM's unified vector store and RAG endpoints. LiteLLM calls the S3 Vectors REST API directly with SigV4-signed requests; no...
- [Valkey - Vector Store](https://docs.litellm.ai/docs/providers/valkey_vector_stores.md): Search documents you have already indexed in Valkey through LiteLLM's unified vector store API, so that any virtual key can run retrieval against your datastore without ever...
- [RAGFlow Vector Stores](https://docs.litellm.ai/docs/providers/ragflow_vector_store.md): Litellm support creation and management of datasets for document processing and knowledge base management in Ragflow.
- [LangGraph](https://docs.litellm.ai/docs/providers/langgraph.md): Call LangGraph agents through LiteLLM using the OpenAI chat completions format.
- [Pydantic AI Agents](https://docs.litellm.ai/docs/providers/pydantic_ai_agent.md): Call Pydantic AI Agents via LiteLLM's A2A Gateway.
- [Abliteration](https://docs.litellm.ai/docs/providers/abliteration.md)
- [AI21](https://docs.litellm.ai/docs/providers/ai21.md): LiteLLM supports the following AI21 models:
- [AI/ML API](https://docs.litellm.ai/docs/providers/aiml.md)
- [Aleph Alpha](https://docs.litellm.ai/docs/providers/aleph_alpha.md): LiteLLM supports all models from Aleph Alpha.
- [Amazon Nova](https://docs.litellm.ai/docs/providers/amazon_nova.md): | Property | Details |
- [Anyscale](https://docs.litellm.ai/docs/providers/anyscale.md)
- [Apertis AI (Stima API)](https://docs.litellm.ai/docs/providers/apertis.md)
- [AWS Sagemaker](https://docs.litellm.ai/docs/providers/aws_sagemaker.md): LiteLLM supports All Sagemaker Huggingface Jumpstart Models
- [Baseten](https://docs.litellm.ai/docs/providers/baseten.md): LiteLLM supports both Baseten Model APIs and dedicated deployments with automatic routing.
- [Black Forest Labs Image Generation](https://docs.litellm.ai/docs/providers/black_forest_labs.md): Black Forest Labs provides state-of-the-art text-to-image generation using their FLUX models.
- [Black Forest Labs Image Editing](https://docs.litellm.ai/docs/providers/black_forest_labs_img_edit.md): Black Forest Labs provides powerful image editing capabilities using their FLUX models to modify existing images based on text descriptions.
- [Bytez](https://docs.litellm.ai/docs/providers/bytez.md): LiteLLM supports all chat models on Bytez!
- [Cerebras](https://docs.litellm.ai/docs/providers/cerebras.md)
- [ChatGPT Subscription](https://docs.litellm.ai/docs/providers/chatgpt.md): Use a ChatGPT subscription that includes Codex (see OpenAI's Codex pricing for the plans, from Plus and Pro to Business and Enterprise) through LiteLLM with OAuth device flow...
- [Chutes](https://docs.litellm.ai/docs/providers/chutes.md)
- [Clarifai](https://docs.litellm.ai/docs/providers/clarifai.md): Anthropic, OpenAI, Qwen, xAI, Gemini and most of Open soured LLMs are Supported on Clarifai.
- [Cloudflare Workers AI](https://docs.litellm.ai/docs/providers/cloudflare_workers.md)
- [Codestral API [Mistral AI]](https://docs.litellm.ai/docs/providers/codestral.md): Codestral is available in select code-completion plugins but can also be queried directly. See the documentation for more details.
- [Cognition](https://docs.litellm.ai/docs/providers/cognition.md)
- [Cohere](https://docs.litellm.ai/docs/providers/cohere.md)
- [CompactifAI](https://docs.litellm.ai/docs/providers/compactifai.md)
- [CoralBricks](https://docs.litellm.ai/docs/providers/coralbricks.md)
- [Crusoe](https://docs.litellm.ai/docs/providers/crusoe.md)
- [Dashscope API (Qwen models)](https://docs.litellm.ai/docs/providers/dashscope.md)
- [Databricks](https://docs.litellm.ai/docs/providers/databricks.md): LiteLLM supports all models on Databricks
- [DataRobot](https://docs.litellm.ai/docs/providers/datarobot.md): LiteLLM supports all models from DataRobot. Select datarobot as the provider to route your request through the datarobot OpenAI-compatible endpoint using the upstream official...
- [Deepgram](https://docs.litellm.ai/docs/providers/deepgram.md): LiteLLM supports Deepgram's /listen endpoint.
- [DeepInfra](https://docs.litellm.ai/docs/providers/deepinfra.md)
- [Eden AI](https://docs.litellm.ai/docs/providers/edenai.md)
- [ElevenLabs](https://docs.litellm.ai/docs/providers/elevenlabs.md): ElevenLabs provides high-quality AI voice technology, including speech-to-text capabilities through their transcription API.
- [EmpirioLabs AI](https://docs.litellm.ai/docs/providers/empiriolabs.md)
- [Fal AI](https://docs.litellm.ai/docs/providers/fal_ai.md): Fal AI provides fast, scalable access to state-of-the-art image generation models including FLUX, Stable Diffusion, Imagen, and more.
- [Featherless AI](https://docs.litellm.ai/docs/providers/featherless_ai.md)
- [Fireworks AI](https://docs.litellm.ai/docs/providers/fireworks_ai.md): We support ALL Fireworks AI models, just set fireworks_ai/ as a prefix when sending completion requests
- [FriendliAI](https://docs.litellm.ai/docs/providers/friendliai.md): We support ALL FriendliAI models, just set friendliai/ as a prefix when sending completion requests
- [Galadriel](https://docs.litellm.ai/docs/providers/galadriel.md)
- [Github](https://docs.litellm.ai/docs/providers/github.md)
- [GitHub Copilot](https://docs.litellm.ai/docs/providers/github_copilot.md): Use GitHub Copilot models through the LiteLLM gateway or the Python SDK. LiteLLM supports Chat Completions, Responses, Embeddings, and Anthropic Messages
- [GMI Cloud](https://docs.litellm.ai/docs/providers/gmi.md)
- [GradientAI](https://docs.litellm.ai/docs/providers/gradient_ai.md)
- [Heroku](https://docs.litellm.ai/docs/providers/heroku.md)
- [Hugging Face](https://docs.litellm.ai/docs/providers/huggingface.md): LiteLLM supports running inference across multiple services for models hosted on the Hugging Face Hub.
- [HuggingFace Rerank](https://docs.litellm.ai/docs/providers/huggingface_rerank.md): HuggingFace Rerank allows you to use reranking models hosted on Hugging Face infrastructure or your custom endpoints to reorder documents based on their relevance to a query.
- [Hyperbolic](https://docs.litellm.ai/docs/providers/hyperbolic.md)
- [Inception](https://docs.litellm.ai/docs/providers/inception.md)
- [Inference APIs](https://docs.litellm.ai/docs/providers/inferenceapis.md)
- [Jina AI](https://docs.litellm.ai/docs/providers/jina_ai.md)
- [Lambda AI](https://docs.litellm.ai/docs/providers/lambda_ai.md)
- [LlamaGate](https://docs.litellm.ai/docs/providers/llamagate.md)
- [Manus](https://docs.litellm.ai/docs/providers/manus.md): Use Manus AI agents through LiteLLM's OpenAI-compatible Responses API.
- [Meta Model API](https://docs.litellm.ai/docs/providers/meta.md): | Property | Details |
- [Meta Llama](https://docs.litellm.ai/docs/providers/meta_llama.md): | Property | Details |
- [Microsoft 365 Copilot](https://docs.litellm.ai/docs/providers/microsoft_365_copilot.md): Connect an app with Microsoft Entra single sign-on (SSO) to your remotely deployed LiteLLM gateway. Users sign in with their Microsoft account, then ask Copilot about the...
- [MiniMax](https://docs.litellm.ai/docs/providers/minimax.md)
- [Moonshot AI](https://docs.litellm.ai/docs/providers/moonshot.md)
- [Morph](https://docs.litellm.ai/docs/providers/morph.md): LiteLLM supports all models on Morph
- [Nadir](https://docs.litellm.ai/docs/providers/nadir.md)
- [NanoGPT](https://docs.litellm.ai/docs/providers/nano-gpt.md)
- [Nebius AI Studio](https://docs.litellm.ai/docs/providers/nebius.md)
- [Neosantara](https://docs.litellm.ai/docs/providers/neosantara.md)
- [NLP Cloud](https://docs.litellm.ai/docs/providers/nlp_cloud.md): LiteLLM supports all LLMs on NLP Cloud.
- [Novita AI](https://docs.litellm.ai/docs/providers/novita.md): | Property | Details |
- [Nscale (EU Sovereign)](https://docs.litellm.ai/docs/providers/nscale.md)
- [Nvidia Riva (Speech-to-Text)](https://docs.litellm.ai/docs/providers/nvidia_riva.md): LiteLLM supports NVIDIA Riva for speech-to-text via /audio/transcriptions. Works with both the NVCF-hosted Riva endpoint (e.g. Parakeet on build.nvidia.com) and self-hosted Riva...
- [Oracle Cloud Infrastructure (OCI)](https://docs.litellm.ai/docs/providers/oci.md): LiteLLM supports the following models for OCI on-demand GenAI API.
- [OpenInfer](https://docs.litellm.ai/docs/providers/openinfer.md)
- [OVHCloud AI Endpoints](https://docs.litellm.ai/docs/providers/ovhcloud.md): Leading French Cloud provider in Europe with data sovereignty and privacy.
- [Perplexity AI (pplx-api)](https://docs.litellm.ai/docs/providers/perplexity.md)
- [Perplexity Embeddings](https://docs.litellm.ai/docs/providers/perplexity_embedding.md)
- [Predibase](https://docs.litellm.ai/docs/providers/predibase.md): LiteLLM supports all models on Predibase
- [Prism](https://docs.litellm.ai/docs/providers/prism.md)
- [PublicAI](https://docs.litellm.ai/docs/providers/publicai.md)
- [QwenCloud and Qianwen AI Platform (Qwen models)](https://docs.litellm.ai/docs/providers/qwencloud.md): Alibaba unified its model platform brands (DashScope, Bailian, Model Studio) under QwenCloud outside mainland China and Qianwen AI Platform in mainland China. LiteLLM routes both...
- [RAGFlow](https://docs.litellm.ai/docs/providers/ragflow.md): Litellm supports Ragflow's chat completions APIs
- [Recraft](https://docs.litellm.ai/docs/providers/recraft.md)
- [Reka](https://docs.litellm.ai/docs/providers/reka.md)
- [Replicate](https://docs.litellm.ai/docs/providers/replicate.md): LiteLLM supports all models on Replicate
- [RunwayML - Image Generation](https://docs.litellm.ai/docs/providers/runwayml/images.md)
- [RunwayML - Video Generation](https://docs.litellm.ai/docs/providers/runwayml/videos.md): LiteLLM supports RunwayML's Gen-4 video generation API, allowing you to generate videos from text prompts and images.
- [SambaNova](https://docs.litellm.ai/docs/providers/sambanova.md)
- [SAP Generative AI Hub](https://docs.litellm.ai/docs/providers/sap.md): LiteLLM supports SAP Generative AI Hub's Orchestration Service.
- [Sarvam.ai](https://docs.litellm.ai/docs/providers/sarvam.md): LiteLLM supports all the text models from Sarvam ai
- [ScaleDown](https://docs.litellm.ai/docs/providers/scaledown.md): Use ScaleDown to classify text, extract fields, summarize documents supplied as text, or compress prompts. LiteLLM calls ScaleDown's native APIs and returns the result through...
- [Scaleway](https://docs.litellm.ai/docs/providers/scaleway.md): LiteLLM supports all models available on Scaleway Generative APIs .
- [SCX.ai](https://docs.litellm.ai/docs/providers/scx_ai.md)
- [Snowflake Cortex](https://docs.litellm.ai/docs/providers/snowflake.md): LiteLLM supports all models on the Snowflake Cortex REST API, including models from Anthropic (Claude), OpenAI (GPT), Meta (Llama), Mistral, DeepSeek, and Snowflake.
- [Stability AI](https://docs.litellm.ai/docs/providers/stability.md)
- [Synthetic](https://docs.litellm.ai/docs/providers/synthetic.md)
- [Tencent TokenHub](https://docs.litellm.ai/docs/providers/tencent.md)
- [Tensormesh](https://docs.litellm.ai/docs/providers/tensormesh.md)
- [Together AI](https://docs.litellm.ai/docs/providers/togetherai.md): LiteLLM supports all models on Together AI.
- [Topaz](https://docs.litellm.ai/docs/providers/topaz.md): | Property | Details |
- [v0](https://docs.litellm.ai/docs/providers/v0.md)
- [Volcano Engine (Volcengine)](https://docs.litellm.ai/docs/providers/volcano.md)
- [VoyageAI by MongoDB](https://docs.litellm.ai/docs/providers/voyage.md)
- [Weights & Biases Inference](https://docs.litellm.ai/docs/providers/wandb_inference.md)
- [IBM watsonx.ai](https://docs.litellm.ai/docs/providers/watsonx/index.md): LiteLLM supports all IBM watsonx.ai foundational models and embeddings.
- [WatsonX Audio Transcription](https://docs.litellm.ai/docs/providers/watsonx/audio_transcription.md)
- [Xiaomi MiMo](https://docs.litellm.ai/docs/providers/xiaomi_mimo.md)
- [Z.AI (Zhipu AI)](https://docs.litellm.ai/docs/providers/zai.md)
- [Integrate as a Model Provider](https://docs.litellm.ai/docs/provider_registration/index.md): Quick Start for OpenAI-Compatible Providers
- [Adding OpenAI-Compatible Providers](https://docs.litellm.ai/docs/contributing/adding_openai_compatible_providers.md): For simple OpenAI-compatible providers (like Hyperbolic, Nscale, etc.), you can add support by editing a single JSON file.
- [Add Model Pricing & Context Window](https://docs.litellm.ai/docs/provider_registration/add_model_pricing.md): To add pricing or context window information for a model, simply make a PR to this file:

## Troubleshooting

- [UI Troubleshooting](https://docs.litellm.ai/docs/troubleshoot/ui_issues.md): If you're experiencing issues with the LiteLLM Admin UI, please include the following information when reporting.
- [Debugging a cost discrepancy](https://docs.litellm.ai/docs/troubleshoot/cost_discrepancy.md): Cost discrepancies between LiteLLM and your provider bill usually come from one of three areas: token ingestion, the cost formula LiteLLM applies, or stale or incorrect pricing...
- [Model missing after Reload Price Data](https://docs.litellm.ai/docs/troubleshoot/missing_model.md): A model that was just added to LiteLLM's pricing map does not show up in the Admin UI model picker, or cost tracking logs This model isn't mapped yet and records $0 spend for it,...
- [Latency Overhead Troubleshooting](https://docs.litellm.ai/docs/troubleshoot/latency_overhead.md): Use this guide when you see unexpected latency overhead between LiteLLM proxy and the LLM provider.
- [CPU Issue Classification & Reproduction](https://docs.litellm.ai/docs/troubleshoot/cpu_issues.md): 1. Classify the CPU Issue
- [Memory Issue Classification & Reproduction](https://docs.litellm.ai/docs/troubleshoot/memory_issues.md): 1. Classify the Memory Issue
- [Spend Update Queue Full Warnings](https://docs.litellm.ai/docs/troubleshoot/spend_queue_warnings.md)
- [MAX_CALLBACKS Limit](https://docs.litellm.ai/docs/troubleshoot/max_callbacks.md)
- [Troubleshooting Prisma Migration Errors](https://docs.litellm.ai/docs/troubleshoot/prisma_migrations.md): Common Prisma migration issues encountered when upgrading or downgrading LiteLLM proxy versions, plus the query engine resolution failure that can stop the proxy before any...
- [Upgrading LiteLLM Proxy (uv/venv)](https://docs.litellm.ai/docs/troubleshoot/pip_venv_upgrade.md): Guide for upgrading LiteLLM Proxy when installed via uv in a virtual environment.
- [Safe Rollback Guide](https://docs.litellm.ai/docs/troubleshoot/rollback.md): This guide outlines the process for safely rolling back a LiteLLM Proxy deployment to a previous version.
- [Outage Troubleshooting](https://docs.litellm.ai/docs/troubleshoot.md): Use this page when LiteLLM nodes go down: pods restart or get OOMKilled, the load balancer marks targets unhealthy, or requests time out across the deployment. Most outages we...

## Resources

- [Custom Pricing - SageMaker, Azure, etc](https://docs.litellm.ai/docs/sdk_custom_pricing.md): Register custom pricing for sagemaker completion model
- [Migration Guide - LiteLLM v1.0.0+](https://docs.litellm.ai/docs/migration.md): When we have breaking changes (i.e. going from 1.x.x to 2.x.x), we will document those changes here.
- [Data Privacy and Security](https://docs.litellm.ai/docs/data_security.md): At LiteLLM, safeguarding your data privacy and security is our top priority. We recognize the critical importance of the data you share with us and handle it with the highest...
- [LiteLLM Self-Hosted Security & Encryption FAQ](https://docs.litellm.ai/docs/proxy/security_encryption_faq.md)
- [Docker Image Security Guide](https://docs.litellm.ai/docs/proxy/docker_image_security.md): LiteLLM signs every Docker image published to GHCR with cosign starting from v1.83.0. This page covers how to verify signatures, enforce verification in CI/CD, and follow...
- [Migration Policy](https://docs.litellm.ai/docs/migration_policy.md)
- [API Stability Policy](https://docs.litellm.ai/docs/api_stability_policy.md): This page defines which parts of the LiteLLM Proxy HTTP API are a public contract we work to preserve, and which parts are private and may change at any time.
- [Shared Responsibility Model](https://docs.litellm.ai/docs/shared_responsibility.md): What LiteLLM is responsible for and what you are responsible for when you self-host the gateway, and how to tell which side an issue falls on.
- [Release Cycle](https://docs.litellm.ai/docs/proxy/release_cycle.md): Litellm Proxy has the following release cycle:
- [LiteLLM Proxy - 1K RPS Load test on locust](https://docs.litellm.ai/docs/load_test_advanced.md): Tutorial on how to get to 1K+ RPS with LiteLLM Proxy on locust
- [LiteLLM SDK vs OpenAI](https://docs.litellm.ai/docs/load_test_sdk.md): Here is a script to load test LiteLLM vs OpenAI
- [Multi-Instance TPM/RPM (litellm.Router)](https://docs.litellm.ai/docs/load_test_rpm.md): Test if your defined tpm/rpm limits are respected across multiple instances of the Router object.
- [Smolagents](https://docs.litellm.ai/docs/projects/smolagents.md): smolagents is a barebones library for agents. Agents write python code to call tools and orchestrate other agents.
- [mini-swe-agent](https://docs.litellm.ai/docs/projects/mini-swe-agent.md): mini-swe-agent The 100 line AI agent that solves GitHub issues & more.
- [OpenAI Agents SDK](https://docs.litellm.ai/docs/projects/openai-agents.md): Use OpenAI Agents SDK with any LLM provider through LiteLLM Proxy.
- [Google ADK (Agent Development Kit)](https://docs.litellm.ai/docs/projects/Google ADK.md): Google ADK is an open-source, code-first Python framework for building, evaluating, and deploying sophisticated AI agents. While optimized for Gemini, ADK is model-agnostic and...
- [Agent Lightning](https://docs.litellm.ai/docs/projects/Agent Lightning.md): Agent Lightning is Microsoft's open-source framework for training and optimizing AI agents with Reinforcement Learning, Automatic Prompt Optimization, and Supervised Fine-tuning,...
- [Harbor](https://docs.litellm.ai/docs/projects/Harbor.md): Harbor is a framework from the creators of Terminal-Bench for evaluating and optimizing agents and language models. It uses LiteLLM to call 100+ LLM providers.
- [CompatCanary](https://docs.litellm.ai/docs/projects/CompatCanary.md): CompatCanary is an open-source conformance scanner for OpenAI-compatible endpoints. It sends deterministic probes through a LiteLLM proxy and records observed behavior for chat...
- [Microsoft GraphRAG](https://docs.litellm.ai/docs/projects/GraphRAG.md): GraphRAG is a data pipeline and transformation suite that extracts meaningful, structured data from unstructured text using LLMs. It uses a graph-based approach to RAG...
- [Docq.AI](https://docs.litellm.ai/docs/projects/Docq.AI.md): A private and secure ChatGPT alternative that knows your business.
- [PDL](https://docs.litellm.ai/docs/projects/PDL.md): PDL - A YAML-based approach to prompt programming
- [OpenInterpreter](https://docs.litellm.ai/docs/projects/OpenInterpreter.md): Open Interpreter lets LLMs run code on your computer to complete tasks.
- [Elroy](https://docs.litellm.ai/docs/projects/Elroy.md): Elroy is a scriptable AI assistant that remembers and sets goals.
- [dbally](https://docs.litellm.ai/docs/projects/dbally.md): Efficient, consistent and secure library for querying structured data with natural language. Query any database with over 100 LLMs .
- [FastREPL](https://docs.litellm.ai/docs/projects/FastREPL.md): Fast Run-Eval-Polish Loop for LLM Applications
- [PROMPTMETHEUS](https://docs.litellm.ai/docs/projects/PROMPTMETHEUS.md): PROMPTMETHEUS – Prompt Engineering IDE
- [Codium PR Agent](https://docs.litellm.ai/docs/projects/Codium PR Agent.md): An AI-Powered Tool for Automated Pull Request Analysis,
- [Prompt2Model](https://docs.litellm.ai/docs/projects/Prompt2Model.md): Prompt2Model - Generate Deployable Models from Instructions
- [SalesGPT](https://docs.litellm.ai/docs/projects/SalesGPT.md): SalesGPT - Your Context-Aware AI Sales Assistant
- [Softgen](https://docs.litellm.ai/docs/projects/Softgen.md): Softgen is an AI-powered platform that builds full-stack web apps from your plain instructions.
- [Quivr](https://docs.litellm.ai/docs/projects/Quivr.md): Your Second Brain supercharged by Generative AI. Dump all your files and chat with your personal assistant on your files & more using GPT 3.5/4, Private, Anthropic, VertexAI,...
- [Langstream](https://docs.litellm.ai/docs/projects/Langstream.md): Build robust LLM applications with true composability
- [Otter](https://docs.litellm.ai/docs/projects/Otter.md): Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following and in-context...
- [YiVal](https://docs.litellm.ai/docs/projects/YiVal.md): Evaluate and Evolve. YiVal is an open source GenAI-Ops framework that allows you to manually or automatically tune and evaluate your AIGC prompts, retrieval configs and fine-tune...
- [llmcord.py](https://docs.litellm.ai/docs/projects/llm_cord.md): llmcord.py lets you and your friends chat with LLMs directly in your Discord server. It works with practically any LLM, remote or locally hosted.
- [pgai](https://docs.litellm.ai/docs/projects/pgai.md): pgai is a suite of tools to develop RAG, semantic search, and other AI applications more easily with PostgreSQL.
- [GPTLocalhost](https://docs.litellm.ai/docs/projects/GPTLocalhost.md): GPTLocalhost - LiteLLM is supported by GPTLocalhost, a local Word Add-in for you to use models in LiteLLM within Microsoft Word. 100% Private.
- [HolmesGPT](https://docs.litellm.ai/docs/projects/HolmesGPT.md): HolmesGPT is an AI-powered observability tool designed to enhance incident response and troubleshooting processes. It's like your 24/7 on-call assistant, helps you solve alerts...
- [Railtracks](https://docs.litellm.ai/docs/projects/Railtracks.md): Railtracks is an open-source agentic framework that helps developers build resilient agentic systems offering local and remote monitoring tools.
- [Code Quality](https://docs.litellm.ai/docs/extras/code_quality.md): LiteLLM follows the Google Python Style Guide.
- [Rules](https://docs.litellm.ai/docs/rules.md): Use this to fail a request based on the input or output of an llm api call.
- [[DEPRECATED] Team-based Routing](https://docs.litellm.ai/docs/proxy/team_based_routing.md): This is deprecated, please use Tag Based Routing instead
- [[DEPRECATED] Region-based Routing](https://docs.litellm.ai/docs/proxy/customer_routing.md): This is deprecated, please use Tag Based Routing instead
- [[OLD PROXY [NEW proxy here](./simple_proxy)] Local LiteLLM Proxy Server](https://docs.litellm.ai/docs/proxy_server.md): A fast, and lightweight OpenAI-compatible server to call 100+ LLM APIs.
- [Contributing Code](https://docs.litellm.ai/docs/extras/contributing_code.md): Checklist before submitting a PR
- [Directory Structure](https://docs.litellm.ai/docs/adding_provider/directory_structure.md): When adding a new provider, you need to create a directory for the provider that follows the following structure:
- [Add Rerank Provider](https://docs.litellm.ai/docs/adding_provider/new_rerank_provider.md): LiteLLM follows the Cohere Rerank API format for all rerank providers. Here's how to add a new rerank provider:
- [Contributing to Documentation](https://docs.litellm.ai/docs/extras/contributing.md): This website is built using Docusaurus 3, a modern static website generator.
- [Contributing - UI](https://docs.litellm.ai/docs/contributing.md): Thanks for contributing to the LiteLLM UI! This guide will help you set up your local development environment.

## Learn: guides and tutorials

- [Learn LiteLLM](https://docs.litellm.ai/docs/learn.md): LiteLLM gives you one OpenAI-compatible interface for 100+ LLM providers. Start with the path that matches your setup.
- [Gateway Quickstart](https://docs.litellm.ai/docs/learn/gateway_quickstart.md): Start LiteLLM Gateway, add models and keys, then connect applications and SDKs to one shared endpoint.
- [SDK Quickstart](https://docs.litellm.ai/docs/learn/python_sdk_quickstart.md): Make your first LiteLLM SDK call, then jump to the right docs for the next feature you need.
- [Enterprise Quickstart](https://docs.litellm.ai/docs/learn/enterprise_quickstart_learn.md): Roll out LiteLLM Enterprise in five steps. Deploy the gateway, give access to models, MCP tools, and agents, connect clients, secure authentication, and allocate costs across...
- [Set up environment](https://docs.litellm.ai/docs/tutorials/installation.md): Let's get the necessary keys to set up our demo environment.
- [Create your first LLM playground](https://docs.litellm.ai/docs/tutorials/first_playground.md): Create a playground to evaluate multiple LLM Providers in less than 10 minutes. If you want to see this in prod, check out our website.
- [Using Text Completion Format - with Completion()](https://docs.litellm.ai/docs/tutorials/text_completion.md): If your prefer interfacing with the OpenAI Text Completion format this tutorial covers how to use LiteLLM in this format
- [Mock Completion Responses - Save Testing Costs](https://docs.litellm.ai/docs/tutorials/mock_completion.md): Trying to test making LLM Completion calls without calling the LLM APIs ?
- [provider_specific_params](https://docs.litellm.ai/docs/tutorials/provider_specific_params.md)
- [Gradio Chatbot + LiteLLM Tutorial](https://docs.litellm.ai/docs/tutorials/gradio_integration.md): Simple tutorial for integrating LiteLLM completion calls with streaming Gradio chatbot demos
- [Replacing OpenAI ChatCompletion with Completion()](https://docs.litellm.ai/docs/tutorials/azure_openai.md)
- [Llama Together AI Tutorial](https://docs.litellm.ai/docs/tutorials/TogetherAI_liteLLM.md)
- [Llama2 - Huggingface Tutorial](https://docs.litellm.ai/docs/tutorials/huggingface_tutorial.md): Huggingface is an open source platform to deploy machine-learnings models.
- [CodeLlama - Code Infilling](https://docs.litellm.ai/docs/tutorials/huggingface_codellama.md): This tutorial shows how you can call CodeLlama (hosted on Huggingface PRO Inference Endpoints), to fill code.
- [Using Fine-Tuned gpt-3.5-turbo](https://docs.litellm.ai/docs/tutorials/finetuned_chat_gpt.md): LiteLLM allows you to call completion with your fine-tuned gpt-3.5-turbo models
- [Oobabooga Text Web API Tutorial](https://docs.litellm.ai/docs/tutorials/oobabooga.md)
- [Onboard Users for AI Exploration](https://docs.litellm.ai/docs/tutorials/default_team_self_serve.md): v1.73.0 introduces the ability to assign new users to Default Teams. This makes it much easier to enable experimentation with LLMs within your company, by allowing users to sign...
- [Microsoft SSO: Sync Groups, Members with LiteLLM](https://docs.litellm.ai/docs/tutorials/msft_sso.md): Sync Microsoft SSO Groups, Members with LiteLLM Teams.
- [[Beta] Routing based on request metadata](https://docs.litellm.ai/docs/tutorials/tag_management.md): Create routing rules based on request metadata.
- [Model Fallbacks (Provider Failover) w/ LiteLLM](https://docs.litellm.ai/docs/tutorials/model_fallbacks.md): Implement model fallbacks (provider failover) across OpenAI, Anthropic, and Azure with LiteLLM so a failing provider fails over to a backup.
- [Using completion() with Fallbacks (Failover) for Reliability](https://docs.litellm.ai/docs/tutorials/fallbacks.md): Use completion() with model fallbacks (failover) so a failing provider automatically fails over to a backup model for reliable responses.
- [Auto-Inject Prompt Caching Checkpoints](https://docs.litellm.ai/docs/tutorials/prompt_caching.md): Reduce costs by up to 90% by using LiteLLM to auto-inject prompt caching checkpoints.
- [Aporia Guardrails with LiteLLM Gateway](https://docs.litellm.ai/docs/tutorials/litellm_proxy_aporia.md): In this tutorial we will use LiteLLM AI Gateway with Aporia to detect PII in requests and profanity in responses
- [Presidio PII Masking with LiteLLM - Complete Tutorial](https://docs.litellm.ai/docs/tutorials/presidio_pii_masking.md): This tutorial will guide you through setting up PII (Personally Identifiable Information) masking with Microsoft Presidio and LiteLLM Gateway. By the end of this tutorial, you'll...
- [File Search in the Responses API](https://docs.litellm.ai/docs/tutorials/file_search_responses_api.md): LiteLLM now supports file_search in the Responses API across both:
- [Chat with sample documents in MongoDB (BETA)](https://docs.litellm.ai/docs/tutorials/mongodb_vector_search.md): Create three fictional policy documents, index them in MongoDB Atlas, test semantic search in the LiteLLM Admin UI, and use the results in a chat completion. The sample text...
- [Using Anthropic File API with LiteLLM Proxy](https://docs.litellm.ai/docs/tutorials/anthropic_file_usage.md)
- [Call Gemini Realtime API with Audio Input/Output](https://docs.litellm.ai/docs/tutorials/gemini_realtime_with_audio.md)
- [Elasticsearch Logging with LiteLLM](https://docs.litellm.ai/docs/tutorials/elasticsearch_logging.md): Send your LLM requests, responses, costs, and performance data to Elasticsearch for analytics and monitoring using OpenTelemetry.
- [Benchmark LLMs](https://docs.litellm.ai/docs/tutorials/compare_llms.md): Easily benchmark LLMs for a given question by viewing
- [Reliability test Multiple LLM Providers with LiteLLM](https://docs.litellm.ai/docs/tutorials/litellm_Test_Multiple_Providers.md)
- [Evaluate LLMs - MLflow Evals, Auto Eval](https://docs.litellm.ai/docs/tutorials/eval_suites.md)
- [Benchmark LLMs - LM Harness, FastEval, Flask](https://docs.litellm.ai/docs/tutorials/lm_evaluation_harness.md)
- [Cookbook](https://docs.litellm.ai/docs/tutorials/index.md): The Cookbook has step-by-step recipes. Each recipe connects LiteLLM to an external tool, framework, or service, or builds a complete workflow from start to end. To set up LiteLLM...

## Integrations

- [Integrations](https://docs.litellm.ai/docs/integrations/index.md): This section covers integrations with various tools and services that can be used with LiteLLM (either Proxy or SDK).
- [Open WebUI](https://docs.litellm.ai/docs/tutorials/openweb_ui.md): Rolling Open WebUI out to a whole company with SSO, IdP groups mapped to LiteLLM teams, skills, and artifacts? See Open WebUI for Organizations.
- [Open WebUI for Organizations (SSO, Teams, Skills, Artifacts)](https://docs.litellm.ai/docs/tutorials/openwebui_org_rollout.md): Roll out Open WebUI on top of LiteLLM as an internal replacement for the Claude and ChatGPT desktop apps, with single sign-on, the signed-in user's IdP token validated by LiteLLM...
- [Claude Code Compatibility](https://docs.litellm.ai/docs/claude_code_compatibility.md): This table is regenerated daily by an automated populator that runs the
- [Claude Code - Cut Costs](https://docs.litellm.ai/docs/tutorials/claude_code_cut_costs.md): Claude Code is one of the heaviest consumers of input tokens in a modern engineering org. Long tool loops, large file reads, and MCP catalogs with hundreds of tools push every...
- [Auto Router with Claude Code and Claude Desktop](https://docs.litellm.ai/docs/tutorials/claude_code_autorouter.md): Route Claude Code and Claude Desktop through a LiteLLM auto router, including gateway discovery's model-name filter and the organization allowlist requirements that decide...
- [Claude Code Quickstart](https://docs.litellm.ai/docs/tutorials/claude_responses_api.md): This tutorial shows how to call Claude models through LiteLLM proxy from Claude Code.
- [Claude Code - Granular Cost Tracking](https://docs.litellm.ai/docs/tutorials/claude_code_customer_tracking.md): Track Claude Code usage by customer or tags using LiteLLM proxy. This enables granular cost attribution for billing, budgeting, and analytics.
- [Claude Code - Budget Status Line](https://docs.litellm.ai/docs/tutorials/claude_code_budget_statusline.md): Show a user's remaining LiteLLM budget in the Claude Code status line. The status line calls /user/info with the user's virtual key and displays the difference between...
- [Claude Code - Prompt Cache Routing](https://docs.litellm.ai/docs/tutorials/claude_code_prompt_cache_routing.md): Claude's Prompt Caching feature helps to optimize API usage through attempting to cache prompts and re-use cached prompts during subsequent API calls. This feature is used by...
- [Claude Code - WebSearch Across All Providers](https://docs.litellm.ai/docs/tutorials/claude_code_websearch.md): Enable Claude Code's web search tool to work with any provider (Bedrock, Azure, Vertex, etc.). LiteLLM automatically intercepts web search requests and executes them server-side.
- [Use Claude Code with MCPs](https://docs.litellm.ai/docs/tutorials/claude_mcp.md): This tutorial shows how to connect MCP servers to Claude Code via LiteLLM Proxy. For endpoint, transport, and credential selection, see the MCP Configuration Reference
- [Use Claude Code with Non-Anthropic Models](https://docs.litellm.ai/docs/tutorials/claude_non_anthropic_models.md): This tutorial shows how to use Claude Code with non-Anthropic models like OpenAI, Gemini, and other LLM providers through LiteLLM proxy.
- [Claude Code Plugin Marketplace (Managed Skills)](https://docs.litellm.ai/docs/tutorials/claude_code_plugin_marketplace.md): LiteLLM AI Gateway acts as a central registry for Claude Code plugins. Admins can govern which plugins are available across the organization, and engineers can discover and...
- [Claude Code - Managing Anthropic Beta Headers](https://docs.litellm.ai/docs/tutorials/claude_code_beta_headers.md): When using Claude Code with LiteLLM and non-Anthropic providers (Bedrock, Azure AI, Vertex AI), you need to ensure that only supported beta headers are sent to each provider....
- [Claude Code - Context Management](https://docs.litellm.ai/docs/claude_code_context_management.md): LiteLLM supports Anthropic's context_management beta natively across all providers - not just Anthropic.
- [Claude Code - Auto Mode on Non-Claude Models](https://docs.litellm.ai/docs/claude_code_auto_mode.md): LiteLLM answers Claude Code's auto mode safety checks for models that cannot answer them on their own.
- [OpenCode Quickstart](https://docs.litellm.ai/docs/tutorials/opencode_integration.md): This tutorial shows how to connect OpenCode to your existing LiteLLM instance and switch between models.
- [OpenClaw + LiteLLM Integration](https://docs.litellm.ai/docs/tutorials/openclaw_integration.md): OpenClaw is a self-hosted AI assistant that connects chat apps (WhatsApp, Telegram, Discord, and more) to LLM providers. By routing OpenClaw through LiteLLM Proxy, you get access...
- [Cursor Integration](https://docs.litellm.ai/docs/tutorials/cursor_integration.md): Route Cursor IDE requests through LiteLLM for unified logging, budget controls, and access to any model. To connect Cursor to MCP tools, use the direct-client URL and headers in...
- [GitHub Copilot / VS Code](https://docs.litellm.ai/docs/tutorials/github_copilot_integration.md): This tutorial shows two ways to use LiteLLM Proxy from VS Code's chat. The LiteLLM extension adds your gateway as a language model provider, so the chat model picker lists the...
- [Gemini CLI](https://docs.litellm.ai/docs/tutorials/litellm_gemini_cli.md): This tutorial shows you how to integrate the Gemini CLI with LiteLLM Proxy, allowing you to route requests through LiteLLM's unified interface.
- [Qwen Code CLI](https://docs.litellm.ai/docs/tutorials/litellm_qwen_code_cli.md): This tutorial shows you how to integrate the Qwen Code CLI with LiteLLM Proxy, allowing you to route requests through LiteLLM's unified interface.
- [Retool Assist](https://docs.litellm.ai/docs/tutorials/retool_assist.md): This guide walks you through connecting Retool Assist to LiteLLM Proxy. Retool Assist uses AI to generate and edit apps from within the Retool app IDE. Using LiteLLM with Retool...
- [Track Usage for Coding Tools](https://docs.litellm.ai/docs/tutorials/cost_tracking_coding.md): Track usage and costs for AI-powered coding tools like Claude Code, Roo Code, Gemini CLI, and OpenAI Codex through LiteLLM.
- [LiteLLM Skills](https://docs.litellm.ai/docs/tutorials/claude_code_skills.md): litellm-skills is a collection of Agent Skills for managing a live LiteLLM proxy. Install them once and any agent that supports the Agent Skills standard (Claude Code, OpenCode,...
- [OpenAI Agents SDK with LiteLLM](https://docs.litellm.ai/docs/tutorials/openai_agents_sdk.md): Use OpenAI's Agents SDK with any LLM provider through LiteLLM Proxy.
- [Claude Agent SDK with LiteLLM](https://docs.litellm.ai/docs/tutorials/claude_agent_sdk.md): Use Anthropic's Claude Agent SDK with any LLM provider through LiteLLM Proxy.
- [CopilotKit SDK with LiteLLM](https://docs.litellm.ai/docs/tutorials/copilotkit_sdk.md): Use CopilotKit SDK with any LLM provider through LiteLLM Proxy.
- [Google ADK with LiteLLM](https://docs.litellm.ai/docs/tutorials/google_adk.md): Use Google ADK with the LiteLLM Python SDK or the LiteLLM Proxy.
- [Google GenAI SDK with LiteLLM](https://docs.litellm.ai/docs/tutorials/google_genai_sdk.md): Use Google's official GenAI SDK (JavaScript/TypeScript and Python) with any LLM provider through LiteLLM Proxy.
- [LiveKit xAI Realtime Voice Agent](https://docs.litellm.ai/docs/tutorials/livekit_xai_realtime.md): Use LiveKit's xAI Grok Voice Agent plugin with LiteLLM Proxy to build low-latency voice AI agents.
- [Letta Integration](https://docs.litellm.ai/docs/integrations/letta.md): Letta (formerly MemGPT) is a framework for building stateful LLM agents with persistent memory. This guide shows how to integrate both LiteLLM SDK and LiteLLM Proxy with Letta so...
- [Scalekit with LiteLLM](https://docs.litellm.ai/docs/tutorials/scalekit_agentkit.md): Add authenticated tool calls to your LiteLLM-powered agents. Scalekit manages OAuth flows, token storage, and API execution for 100+ third-party apps (Gmail, GitHub, Slack,...
- [Using ChatLiteLLM() - Langchain](https://docs.litellm.ai/docs/langchain/index.md)
- [Instructor](https://docs.litellm.ai/docs/tutorials/instructor.md): Combine LiteLLM with jxnl's instructor library for more robust structured outputs. Outputs are automatically validated into Pydantic types and validation errors are provided back...
- [Observability](https://docs.litellm.ai/docs/integrations/observability_integrations.md): Track, debug, and analyze LLM calls with observability platforms.
- [Migrating to OpenTelemetry v2](https://docs.litellm.ai/docs/observability/opentelemetry_v2_migration.md): OpenTelemetry v2 is a rewrite of LiteLLM's tracing. The HTTP server span is owned by the standard FastAPI instrumentation rather than by LiteLLM, the span model is typed, and...
- [OpenTelemetry v2](https://docs.litellm.ai/docs/observability/opentelemetry_v2.md): OpenTelemetry v2 (OTel v2) is LiteLLM Proxy's next-generation tracing. It gives you one clean trace per request covering the incoming HTTP call, authentication, guardrails, the...
- [OpenTelemetry v1](https://docs.litellm.ai/docs/observability/opentelemetry_integration.md): OpenTelemetry is a CNCF standard for observability. It connects to any observability tool, such as Jaeger, Zipkin, Datadog, New Relic, Traceloop, Levo AI and others.
- [SigNoz](https://docs.litellm.ai/docs/observability/signoz.md): Send LiteLLM traces to SigNoz, Cloud or self-hosted, with the signoz callback. One endpoint field covers both deployments, and under OpenTelemetry v2 each team or virtual key can...
- [AgentOps](https://docs.litellm.ai/docs/observability/agentops_integration.md): Observability and DevTool platform for AI agents, at agentops.ai.
- [Argilla](https://docs.litellm.ai/docs/observability/argilla.md): Argilla is a collaborative annotation tool for AI engineers and domain experts who need to build high-quality datasets for their projects.
- [Arize AX](https://docs.litellm.ai/docs/observability/arize_integration.md): Use Arize AX when you want the full-featured Arize AI platform for production LLM observability and evaluation, available as managed cloud or enterprise self-hosted deployment.
- [Arize Phoenix](https://docs.litellm.ai/docs/observability/phoenix_integration.md): Arize Phoenix is the open-source LLM tracing and evaluation project from Arize AI. Use it for local development, experimentation, and self-hosted workflows.
- [Athina](https://docs.litellm.ai/docs/observability/athina_integration.md): This is community maintained, Please make an issue if you run into a bug
- [Braintrust](https://docs.litellm.ai/docs/observability/braintrust.md): Braintrust manages evaluations, logging, prompt playground, to data management for AI products.
- [Grafana Cloud](https://docs.litellm.ai/docs/observability/grafana_cloud.md): Send LiteLLM traces and GenAI metrics to Grafana Cloud over OTLP
- [Comet Opik](https://docs.litellm.ai/docs/observability/opik_integration.md): Opik is an open source end-to-end LLM Evaluation Platform that helps developers track their LLM prompts and responses during both development and production. Users can define and...
- [DeepEval](https://docs.litellm.ai/docs/observability/deepeval_integration.md)
- [Helicone](https://docs.litellm.ai/docs/observability/helicone_integration.md): This is community maintained. Please make an issue if you run into a bug:
- [Humanloop](https://docs.litellm.ai/docs/observability/humanloop.md): Humanloop enables product teams to build robust AI features with LLMs, using best-in-class tooling for Evaluation, Prompt Management, and Observability.
- [Langfuse](https://docs.litellm.ai/docs/observability/langfuse_integration.md)
- [LangSmith](https://docs.litellm.ai/docs/observability/langsmith_integration.md): An all-in-one developer platform for every step of the application lifecycle
- [Langtrace](https://docs.litellm.ai/docs/observability/langtrace_integration.md): Open-source observability and evaluations for LLM applications, at langtrace.ai.
- [Levo](https://docs.litellm.ai/docs/observability/levo_integration.md): API and LLM security testing and observability, at levo.ai.
- [Literal AI](https://docs.litellm.ai/docs/observability/literalai_integration.md): Literal AI is a collaborative observability, evaluation and analytics platform for building production-grade LLM apps.
- [Logfire](https://docs.litellm.ai/docs/observability/logfire_integration.md): Logfire is open Source Observability & Analytics for LLM Apps
- [Lunary](https://docs.litellm.ai/docs/observability/lunary_integration.md): Lunary is an open-source platform providing observability, prompt management, and analytics to help team manage and improve LLM chatbots.
- [MLflow](https://docs.litellm.ai/docs/observability/mlflow.md)
- [PromptLayer](https://docs.litellm.ai/docs/observability/promptlayer_integration.md): This is community maintained, Please make an issue if you run into a bug
- [Qualifire](https://docs.litellm.ai/docs/observability/qualifire_integration.md): Qualifire provides real-time Agentic evaluations, guardrails and observability for production AI applications.
- [Weights & Biases](https://docs.litellm.ai/docs/observability/wandb_integration.md): This is community maintained, Please make an issue if you run into a bug
- [Azure Sentinel](https://docs.litellm.ai/docs/observability/azure_sentinel.md): LiteLLM supports logging to Azure Sentinel via the Azure Monitor Logs Ingestion API. Azure Sentinel uses Log Analytics workspaces for data storage, so logs sent to the workspace...
- [Datadog](https://docs.litellm.ai/docs/observability/datadog.md): LiteLLM Supports logging to the following Datdog Integrations:
- [Google Cloud Storage](https://docs.litellm.ai/docs/observability/gcs_bucket_integration.md): Log LLM Logs to Google Cloud Storage Buckets
- [New Relic](https://docs.litellm.ai/docs/observability/newrelic.md)
- [Parseable](https://docs.litellm.ai/docs/observability/parseable.md): Parseable provides setup guides for the LiteLLM SDK and LiteLLM Gateway.
- [PostHog](https://docs.litellm.ai/docs/observability/posthog_integration.md)
- [Sentry](https://docs.litellm.ai/docs/observability/sentry.md): This is community maintained, Please make an issue if you run into a bug
- [Slack](https://docs.litellm.ai/docs/observability/slack_integration.md): We want to learn how we can make the callbacks better! Meet the LiteLLM founders or
- [Splunk Observability Cloud](https://docs.litellm.ai/docs/observability/splunk_observability_cloud.md): Send LiteLLM traces to Splunk Observability Cloud using the built-in otel callback and standard OpenTelemetry OTLP environment variables.
- [Sumo Logic](https://docs.litellm.ai/docs/observability/sumologic_integration.md): Send LiteLLM logs to Sumo Logic for observability, monitoring, and analysis.
- [Supabase](https://docs.litellm.ai/docs/observability/supabase_integration.md): This is community maintained, Please make an issue if you run into a bug
- [Databricks Zerobus](https://docs.litellm.ai/docs/observability/zerobus.md): Send LiteLLM Gateway request logs to a Unity Catalog Delta table with Databricks Zerobus Ingest. Query model usage, latency, cost, and request metadata in Databricks, alongside...
- [Callbacks](https://docs.litellm.ai/docs/observability/callbacks.md): Use Callbacks to send Output Data to Posthog, Sentry etc
- [Custom Callbacks](https://docs.litellm.ai/docs/observability/custom_callback.md)
- [Generic API](https://docs.litellm.ai/docs/observability/generic_api.md): Send LiteLLM logs to any HTTP endpoint.
- [Raw Request/Response Logging](https://docs.litellm.ai/docs/observability/raw_request_response.md)
- [Scrub Logged Data](https://docs.litellm.ai/docs/observability/scrub_data.md): Redact messages / mask PII before sending data to logging integrations (langfuse/etc.).
- [CloudZero](https://docs.litellm.ai/docs/observability/cloudzero.md): LiteLLM provides an integration with CloudZero's AnyCost API, allowing you to export your LLM usage data to CloudZero for cost tracking analysis.
- [Focus Export (Experimental)](https://docs.litellm.ai/docs/observability/focus.md): Focus Format export is under active development and currently considered experimental.
- [Greenscale](https://docs.litellm.ai/docs/observability/greenscale_integration.md): This is community maintained, Please make an issue if you run into a bug
- [Lago](https://docs.litellm.ai/docs/observability/lago.md): Lago offers a self-hosted and cloud, metering and usage-based billing solution.
- [Mavvrik](https://docs.litellm.ai/docs/observability/mavvrik.md): LiteLLM can export proxy spend data to Mavvrik as FOCUS 1.2 formatted cost reports. This lets you track and analyse LLM spend within the Mavvrik cost management platform.
- [OpenMeter](https://docs.litellm.ai/docs/observability/openmeter.md): OpenMeter is an Open Source Usage-Based Billing solution for AI/Cloud applications. It integrates with Stripe for easy billing.
- [PointFive](https://docs.litellm.ai/docs/observability/pointfive.md): LiteLLM can ship proxy request logs to PointFive for AI cost and usage analysis. Logs are buffered and uploaded through a presigned URL that the PointFive API issues per batch,...
- [Ramp](https://docs.litellm.ai/docs/observability/ramp_integration.md): Send AI usage and cost data to Ramp for automated spend tracking.
- [Ternary](https://docs.litellm.ai/docs/observability/ternary.md): LiteLLM can export proxy spend data to Ternary as FOCUS 1.2 formatted cost reports. This lets you allocate and analyse LLM spend alongside your cloud costs in Ternary's FinOps...
- [Vantage](https://docs.litellm.ai/docs/observability/vantage.md): LiteLLM can export proxy spend data to Vantage as FOCUS 1.2 formatted cost reports. This lets you visualize LLM spend alongside your cloud infrastructure costs in the Vantage...
- [Aim Security](https://docs.litellm.ai/docs/proxy/guardrails/aim_security.md)
- [Airia](https://docs.litellm.ai/docs/proxy/guardrails/airia.md): The Airia guardrail lets your LiteLLM proxy enforce the guardrails you set up in Airia: prompts are checked before they reach the model, responses before they reach the caller,...
- [Akto](https://docs.litellm.ai/docs/proxy/guardrails/akto.md): Use Akto as a guardrail provider, enabling runtime security for all LLM traffic routed through the proxy. Akto is purpose-built to secure autonomous and agentic AI systems that...
- [Alice](https://docs.litellm.ai/docs/proxy/guardrails/alice.md): The Alice guardrail screens prompts and model responses against policies you configure in Alice. On every call it forwards the request to Alice, which decides what in the payload...
- [Aporia](https://docs.litellm.ai/docs/proxy/guardrails/aporia_api.md): Use Aporia to detect PII in requests and profanity in responses
- [Azure Content Safety Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/azure_content_guardrail.md): LiteLLM supports Azure Content Safety guardrails via the Azure Content Safety API.
- [Bedrock Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/bedrock.md): If you haven't set up or authenticated your Bedrock provider yet, see the Bedrock Provider Setup & Authentication Guide.
- [Context Compression (Compresr)](https://docs.litellm.ai/docs/proxy/guardrails/compresr.md): Compresr compresses LLM context against the question being asked. It keeps the tokens that matter for the answer and drops the rest. As a LiteLLM guardrail, it compresses tool...
- [Conduct Guard](https://docs.litellm.ai/docs/proxy/guardrails/conduct.md): The Conduct guardrail sends each prompt to your Conduct workspace before the model is called. Conduct evaluates the user text against the rules configured for the tool the...
- [CrowdStrike AIDR](https://docs.litellm.ai/docs/proxy/guardrails/crowdstrike_aidr.md): The CrowdStrike AIDR guardrail uses configurable detection policies to identify
- [Custom Code Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/custom_code_guardrail.md): Write custom guardrail logic using Python-like code that runs in a sandboxed environment.
- [Custom Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/custom_guardrail.md): Use this if you want to write code to run a custom guardrail
- [Decision Model Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/decision_model.md)
- [DynamoAI Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/dynamoai.md): LiteLLM supports DynamoAI guardrails for content moderation and policy enforcement on LLM inputs and outputs.
- [EnkryptAI Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/enkryptai.md): LiteLLM supports EnkryptAI guardrails for content moderation and safety checks on LLM inputs and outputs.
- [Gray Swan Cygnal Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/grayswan.md): Use Gray Swan Cygnal to continuously monitor conversations for policy violations, indirect prompt injection (IPI), jailbreak attempts, and other safety risks.
- [Guardrails AI](https://docs.litellm.ai/docs/proxy/guardrails/guardrails_ai.md): Use Guardrails AI (guardrailsai.com) to add checks to LLM output.
- [HiddenLayer Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/hiddenlayer.md): LiteLLM ships with a native integration for HiddenLayer. The proxy sends every request/response to HiddenLayer’s /detection/v1/interactions endpoint so you can block or redact...
- [IBM Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/ibm_guardrails.md): LiteLLM works with IBM's FMS Guardrails for content safety. You can use it to detect jailbreaks, PII, hate speech, and more.
- [Javelin Guardrails](https://docs.litellm.ai/docs/proxy/guardrails/javelin.md): Javelin provides AI safety and content moderation services with support for prompt injection detection, trust & safety violations, and language detection.
- [Lakera AI](https://docs.litellm.ai/docs/proxy/guardrails/lakera_ai.md): Supported endpoints: The Lakera v2 integration only supports the chat completions endpoint (/v1/chat/completions). It is not supported for the Responses API, /v1/messages, MCP,...
- [Lasso Security](https://docs.litellm.ai/docs/proxy/guardrails/lasso_security.md): Use Lasso Security to protect your LLM applications from prompt injection attacks, harmful content generation, and other security threats through input and output validation.
- [LLM-as-a-Judge](https://docs.litellm.ai/docs/proxy/guardrails/llm_as_a_judge.md)
- [LLM Shield Proxy](https://docs.litellm.ai/docs/proxy/guardrails/llm_shield_proxy.md): The LLM Shield Proxy guardrail replaces personal data in each request with realistic stand-in values before it reaches the model, then puts the original values back into the...
- [Microsoft Agent 365 Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/microsoft_agent_365.md): Sends every MCP tool call to the Microsoft Agent 365 evaluation API before LiteLLM runs it. Microsoft Defender returns allow or block, and the call is recorded on the Microsoft...
- [Microsoft Purview Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/microsoft_purview.md): LiteLLM supports Microsoft Purview DLP policies via the Microsoft Graph processContent API.
- [Google Cloud Model Armor](https://docs.litellm.ai/docs/proxy/guardrails/model_armor.md): LiteLLM supports Google Cloud Model Armor guardrails via the Model Armor API.
- [Noma Security](https://docs.litellm.ai/docs/proxy/guardrails/noma_security.md): Use Noma Security to protect your LLM applications with AI content moderation and safety guardrails.
- [Onyx Security](https://docs.litellm.ai/docs/proxy/guardrails/onyx_security.md)
- [OpenAI Moderation](https://docs.litellm.ai/docs/proxy/guardrails/openai_moderation.md)
- [Pangea](https://docs.litellm.ai/docs/proxy/guardrails/pangea.md): The Pangea guardrail uses configurable detection policies (called recipes) from its AI Guard service to identify and mitigate risks in AI application traffic, including:
- [PANW Prisma AIRS](https://docs.litellm.ai/docs/proxy/guardrails/panw_prisma_airs.md): LiteLLM supports PANW Prisma AIRS (AI Runtime Security) guardrails via the Prisma AIRS Scan API. This integration provides Security-as-Code for AI applications using Palo Alto...
- [PII, PHI Masking - Presidio](https://docs.litellm.ai/docs/proxy/guardrails/pii_masking_v2.md)
- [Pillar Security](https://docs.litellm.ai/docs/proxy/guardrails/pillar_security.md): Pillar Security integrates with LiteLLM Proxy via the Generic Guardrail API, providing AI security scanning for your LLM applications.
- [In-memory Prompt Injection Detection](https://docs.litellm.ai/docs/proxy/guardrails/prompt_injection.md): LiteLLM Supports the following methods for detecting prompt injection attacks
- [PromptGuard](https://docs.litellm.ai/docs/proxy/guardrails/promptguard.md): Use PromptGuard to protect your LLM applications with prompt injection detection, PII redaction, topic filtering, entity blocklists, and hallucination detection. PromptGuard is...
- [Qostodian Nexus by Qohash](https://docs.litellm.ai/docs/proxy/guardrails/qohash_qostodian_nexus.md): Qohash is a pioneer of zero-copy data security, the only model designed to secure petabytes of unstructured data in large enterprises. Enterprises run dozens of AI models,...
- [Qualifire](https://docs.litellm.ai/docs/proxy/guardrails/qualifire.md): Use Qualifire to evaluate LLM outputs for quality, safety, and reliability. Detect prompt injections, hallucinations, PII, harmful content, and validate that your AI follows...
- [RepelloAI Argus](https://docs.litellm.ai/docs/proxy/guardrails/repelloai.md): Use RepelloAI Argus to scan prompts and responses against the policies you configure per asset in the Repello dashboard. Argus is a cloud-hosted API; prompts are scanned on...
- [Rubrik Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/rubrik.md): Use Rubrik's moderation and logging integration to screen prompts and responses against an external policy service and batch-log all LLM requests/responses.
- [Secret Detection/Redaction (Enterprise-only)](https://docs.litellm.ai/docs/proxy/guardrails/secret_detection.md): Use this to REDACT API Keys, Secrets sent in requests to an LLM.
- [Sensitive Data Routing (Built-in Guardrail)](https://docs.litellm.ai/docs/proxy/guardrails/sensitive_data_routing.md): Built-in guardrail that detects sensitive data in a request and reroutes it to an on-premise model instead of blocking or redacting it. No external dependencies required.
- [Straiker](https://docs.litellm.ai/docs/proxy/guardrails/straiker.md): The Straiker guardrail applies runtime AI security to traffic routed through LiteLLM. On every call it inspects the prompt and the response, including tool definitions and tool...
- [LiteLLM Tool Permission Guardrail](https://docs.litellm.ai/docs/proxy/guardrails/tool_permission.md): LiteLLM provides the LiteLLM Tool Permission Guardrail that lets you control which tool calls a model is allowed to invoke, using configurable allow/deny rules. This offers...
- [Relevance-Based Compaction (TypeSafe / Jev)](https://docs.litellm.ai/docs/proxy/guardrails/typesafe.md): TypeSafe's Jev model judges whether each completed tool exchange is still relevant to the current task. As a LiteLLM guardrail, it blanks out tool results Jev scores below a...
- [Vigil Guard](https://docs.litellm.ai/docs/proxy/guardrails/vigil_guard.md): Use Vigil Guard as a LiteLLM proxy guardrail to evaluate chat input and model output before it is returned to your application.
- [Wingback](https://docs.litellm.ai/docs/proxy/guardrails/wingback.md): Wingback runtime security integrates with LiteLLM Proxy as a first-class guardrail provider. On every call it inspects prompts and responses through the Wingback connectors...
- [XecGuard](https://docs.litellm.ai/docs/proxy/guardrails/xecguard.md): Use XecGuard (CyCraft) to protect your LLM applications with multi-policy scanning (prompt injection, harmful content, PII, system-prompt enforcement, skills protection) and RAG...
- [Zscaler AI Guard](https://docs.litellm.ai/docs/proxy/guardrails/zscaler_ai_guard.md)
- [Alerting / Webhooks](https://docs.litellm.ai/docs/proxy/alerting.md)
- [PagerDuty Alerting](https://docs.litellm.ai/docs/proxy/pagerduty.md): Handles two types of alerts:
- [Prometheus metrics](https://docs.litellm.ai/docs/proxy/prometheus.md): LiteLLM Exposes a /metrics endpoint for Prometheus to Poll
- [Grafana Pyroscope CPU profiling](https://docs.litellm.ai/docs/proxy/pyroscope_profiling.md): LiteLLM proxy can send continuous CPU profiles to Grafana Pyroscope when enabled via environment variables. This is optional and off by default.
- [LiteLLM AI Gateway Prompt Management](https://docs.litellm.ai/docs/proxy/litellm_prompt_management.md): Use the LiteLLM AI Gateway to create, manage and version your prompts.
- [Custom Prompt Management](https://docs.litellm.ai/docs/proxy/custom_prompt_management.md): Connect LiteLLM to your prompt management system with custom hooks.
- [LiteLLM Prompt Management (GitOps)](https://docs.litellm.ai/docs/proxy/native_litellm_prompt.md): Store prompts as .prompt files in your repository and use them directly with LiteLLM. No external services required.
- [Prompt Management](https://docs.litellm.ai/docs/proxy/prompt_management.md): Run experiments or change the specific model (e.g. from gpt-5.6-terra to a gpt-5.6-luna finetune) from your prompt management tool (e.g. Langfuse) instead of making changes in...
- [Arize Phoenix Prompt Management](https://docs.litellm.ai/docs/proxy/arize_phoenix_prompts.md): Use prompt versions from Arize Phoenix with LiteLLM SDK and Proxy.
- [Be an Integration Partner](https://docs.litellm.ai/docs/integrations/community.md)
- [Contribute Custom Webhook API](https://docs.litellm.ai/docs/contribute_integration/custom_webhook_api.md): If your API just needs a Webhook event from LiteLLM, here's how to add a 'native' integration for it on LiteLLM:
- [[BETA] Generic Prompt Management API - Integrate Without a PR](https://docs.litellm.ai/docs/adding_provider/generic_prompt_management_api.md)

## Production rollout

- [Production rollout](https://docs.litellm.ai/docs/learn/enterprise_quickstart.md): Roll out LiteLLM Enterprise in five steps. Deploy the gateway, give access to models, MCP tools, and agents, connect clients, secure authentication, and allocate costs across...

## Activate your license

- [Activate your license](https://docs.litellm.ai/docs/enterprise/activate.md): Set LITELLM_LICENSE on each LiteLLM gateway instance, confirm that the gateway shows Enterprise Edition, and see which features the license key enables.

## Moving from OSS

- [Moving from OSS](https://docs.litellm.ai/docs/enterprise/moving_from_oss.md): Move an open-source LiteLLM gateway to LiteLLM Enterprise. The image, config.yaml, and version line stay the same. Use this checklist before you open the gateway to all your teams.

## Compliance and SOC 2 Type II

- [Compliance and SOC 2 Type II](https://docs.litellm.ai/docs/enterprise/compliance.md): LiteLLM is SOC 2 Type II audited. Get the report from the Trust Center, and find the answers for your security review about data handling, signed images, and vulnerability...

## Support and SLA

- [Support and SLA](https://docs.litellm.ai/docs/enterprise/support.md): LiteLLM Enterprise support. Each license includes a dedicated Slack or Teams channel with the LiteLLM engineers. 24/7 response SLAs and custom SLAs are available.

## Version support

- [Version support](https://docs.litellm.ai/docs/enterprise/version_support.md): LiteLLM supports the four most recent stable minor lines with patch releases. Enterprise and open source use one image and one version number.

## Blog

- [All blog posts](https://docs.litellm.ai/blog.md): every post, newest first
- [Day 0 Support: TypeSafe Jev 1.14: Adds images, box questions, and long context](https://docs.litellm.ai/blog/typesafe_jev_images.md) (2026-10-15): TypeSafe Jev 1.14 reads images, answers box questions with coordinates and takes long-context requests. LiteLLM serves all three on /v1/systemone and...
- [Decision models on LiteLLM: /v1/decisions and /v1/systemone](https://docs.litellm.ai/blog/decision-models.md) (2026-10-14): Decision models return probabilities instead of text, which makes routing, triage and guardrail checks cheap and deterministic. LiteLLM serves nine providers...
- [Workshop: Build a self-hosted cloud agent.](https://docs.litellm.ai/blog/cloud-agent-workshop.md) (2026-10-13): Build a self-hosted cloud agent with the LiteLLM team. A two-week remote workshop, October 19–30, to build a working v0 and deploy an internal beta on your...
- [Gemini 4 Argon API Pricing and Day 0 Support on LiteLLM](https://docs.litellm.ai/blog/gemini-4-argon.md) (2026-10-13): Gemini 4 Argon costs $2.00 per 1M input tokens and $10.00 per 1M output tokens until Nov 30. LiteLLM supports it on day 0 on AI Studio and Google Cloud.
- [October Townhall: Product + Roadmap Updates](https://docs.litellm.ai/blog/october-townhall-announcement.md) (2026-10-10): Join the LiteLLM October townhall on Thursday, 22 October at 7:30 AM PT to learn about LiteLLM's product updates and roadmap.
- [Introducing Microsoft 365 Copilot in LiteLLM](https://docs.litellm.ai/blog/microsoft-365-copilot.md) (2026-10-09): Use Microsoft 365 Copilot from your applications through LiteLLM, with access to your Microsoft 365 work data.
- [Introducing litellm-core: the first step toward a leaner SDK](https://docs.litellm.ai/blog/introducing-litellm-core.md) (2026-10-08): Ongoing work to reduce LiteLLM's dependencies and installation footprint, improve runtime loading, and make imports faster
- [Claude Haiku 5.5 API Pricing and Day 0 Support on LiteLLM](https://docs.litellm.ai/blog/claude-haiku-5-5.md) (2026-10-07): Claude Haiku 5.5 costs $0.10 per 1M input tokens and $0.50 per 1M output tokens. LiteLLM supports it on day 0 on Anthropic, AWS, Google Cloud and Azure.
- [Open Sourcing Moyai: Self-Hosted Cloud Coding Agent](https://docs.litellm.ai/blog/moyai-open-source.md) (2026-10-07): Run cloud coding agents on your own infrastructure with Moyai. Choose your agent and models through LiteLLM, track costs, and follow the setup guide.
- [Day 0 Support: Nano Banana 2.1](https://docs.litellm.ai/blog/nano_banana_2_1.md) (2026-10-06): Day 0 support for Gemini Nano Banana 2.1 on LiteLLM, on Google AI Studio and Gemini Enterprise Agent Platform, with image output at half Nano Banana 2's price.
- [How LiteLLM Lens finds repeated failures across 1,000s of agent traces](https://docs.litellm.ai/blog/lens-failure-patterns.md) (2026-10-05): Inside Lens: parallel trace review, Python tools for large traces, and investigations that connect repeated failures to source evidence.
- [How we cut time to first byte by 94% for long prompts](https://docs.litellm.ai/blog/long-prompt-latency.md) (2026-10-03): A 440k-token benchmark went from 553 ms to 35 ms median time to first byte. We removed unnecessary prompt-cache routing work before the model call.
- [How we built our own internal Devin in 2 days](https://docs.litellm.ai/blog/internal-devin-two-days.md) (2026-10-03): How we built Moyai Devin with Render, Modal, Hermes, Temporal, and LiteLLM: durable sessions, parallel agents, Slack, and shared organization connections.
- [Adding Self-hosted Auto Router Classifiers: Laya & Nimble](https://docs.litellm.ai/blog/laya-nimble-classifiers.md) (2026-10-02): Use Laya or Bespoke Nimble to classify Auto Router requests on your own infrastructure. Control where classification runs, which model you serve, and how you...
- [Launching LiteLLM Lens](https://docs.litellm.ai/blog/litellm-lens-launch.md) (2026-10-01): LiteLLM Lens turns the traces flowing through your gateway into findings your agents can act on. Built for agent swarms generating 200K+ traces.

## Optional

- [Release notes](https://docs.litellm.ai/release_notes): Changelog for every LiteLLM release
- [LiteLLM v1.71.1 Benchmarks](https://docs.litellm.ai/docs/aiohttp_benchmarks.md)
- [Hosted Cache - api.litellm.ai (removed)](https://docs.litellm.ai/docs/caching/caching_api.md): The hosted cache backed by api.litellm.ai has been removed from LiteLLM. "hosted" is not a valid Cache(type=...) value, and passing it leaves the cache without a backend, so the...
- [LiteLLM - Local Caching](https://docs.litellm.ai/docs/caching/local_caching.md): Caching completion() and embedding() calls when switched on
- [Multiple Deployments](https://docs.litellm.ai/docs/completion/multiple_deployments.md): If you have multiple deployments of the same model, you can pass the list of deployments, and LiteLLM will return the first result.
- [Shared Session Support](https://docs.litellm.ai/docs/completion/shared_session.md)
- [Contact Us](https://docs.litellm.ai/docs/contact.md): Have a question, need help getting set up, or want to talk about pricing? Reach out through any of the channels below and we'll get back to you.
- [hosted_debugging](https://docs.litellm.ai/docs/debugging/hosted_debugging.md)
- [Local Debugging](https://docs.litellm.ai/docs/debugging/local_debugging.md): There's 2 ways to do local debugging - litellm.turnondebug() and by passing in a custom function completion(...loggerfn=). Warning: Make sure to not use turnon_debug() in...
- [Get Started](https://docs.litellm.ai/docs/default_code_snippet.md): This section assumes you've already added your API keys in
- [litellm.aembedding()](https://docs.litellm.ai/docs/embedding/async_embedding.md): LiteLLM provides an asynchronous version of the embedding function called aembedding
- [litellm.moderation()](https://docs.litellm.ai/docs/embedding/moderation.md): LiteLLM supports the moderation endpoint for OpenAI
- [Gemini Image Generation Migration Guide](https://docs.litellm.ai/docs/extras/gemini_img_migration.md): Who is impacted by this change?
- [LiteLLM Proxy - Locust Load Test](https://docs.litellm.ai/docs/load_test.md): Locust Load Test LiteLLM Proxy
- [[Beta] Guardrails](https://docs.litellm.ai/docs/old_guardrails.md): Setup Prompt Injection Detection, Secret Detection on LiteLLM Proxy
- [GPT Migrate](https://docs.litellm.ai/docs/projects/GPT Migrate.md): Easily migrate your codebase from one framework or language to another.
- [LiteLLM Proxy](https://docs.litellm.ai/docs/projects/LiteLLM Proxy.md)
- [Anthropic Effort Parameter](https://docs.litellm.ai/docs/providers/anthropic_effort.md): Control how many tokens Claude uses when responding with the effort parameter, trading off between response thoroughness and token efficiency.
- [Anthropic Programmatic Tool Calling](https://docs.litellm.ai/docs/providers/anthropic_programmatic_tool_calling.md): Programmatic tool calling allows Claude to write code that calls your tools programmatically within a code execution container, rather than requiring round trips through the...
- [Anthropic Tool Input Examples](https://docs.litellm.ai/docs/providers/anthropic_tool_input_examples.md): Provide concrete examples of valid tool inputs to help Claude understand how to use your tools more effectively. This is particularly useful for complex tools with nested...
- [Azure AI Image Editing](https://docs.litellm.ai/docs/providers/azure_ai_img_edit.md): Azure AI provides powerful image editing capabilities using FLUX models from Black Forest Labs to modify existing images based on text descriptions.
- [Azure Anthropic (Claude via Azure Foundry)](https://docs.litellm.ai/docs/providers/azure/azure_anthropic.md): LiteLLM supports Claude models deployed via Microsoft Azure Foundry, including Claude Sonnet 4.5, Claude Haiku 4.5, and Claude Opus 4.1.
- [Empower](https://docs.litellm.ai/docs/providers/empower.md): LiteLLM supports all models on Empower.
- [GigaChat](https://docs.litellm.ai/docs/providers/gigachat.md)
- [RunwayML - Text-to-Speech](https://docs.litellm.ai/docs/providers/runwayml/text-to-speech.md)
- [watsonx.ai Rerank](https://docs.litellm.ai/docs/providers/watsonx/rerank.md)
- [LiteLLM Keys (community key, discontinued)](https://docs.litellm.ai/docs/proxy_api.md): The free community sk-litellm-... keys and the hosted proxy behind them are no longer available. The LiteLLM SDK has no special handling for these keys, so setting OPENAIAPIKEY...
- [Credential Usage Tracking](https://docs.litellm.ai/docs/proxy/credential_usage_tracking.md): When a model is attached to a reusable credential, LiteLLM automatically injects the credential name as a tag on every request that uses that model. This means credential-level...
- [Embeddings - /embeddings](https://docs.litellm.ai/docs/proxy/embedding.md): See supported Embedding Providers & Models here
- [Cato Networks](https://docs.litellm.ai/docs/proxy/guardrails/cato_networks.md)
- [Prompt Security](https://docs.litellm.ai/docs/proxy/guardrails/prompt_security.md): Use Prompt Security to protect your LLM applications from prompt injection attacks, jailbreaks, harmful content, PII leakage, and malicious file uploads through input and output...
- [GET Daily Spend, Usage Metrics](https://docs.litellm.ai/docs/proxy/metrics.md): The GET /dailymetrics endpoint that this page used to describe has been removed from the proxy. Daily spend and usage is now served by GET /user/daily/activity, which returns...
- [LiteLLM Proxy Performance](https://docs.litellm.ai/docs/proxy/perf.md): The numbers on this page compare the proxy against calling a provider directly. For gateway capacity numbers (requests, tokens, and latency per pod at scale) see Benchmarks.
- [Reject Client-Side Metadata Tags](https://docs.litellm.ai/docs/proxy/reject_clientside_metadata_tags.md)
- [Shared Health Check State Across Pods](https://docs.litellm.ai/docs/proxy/shared_health_check.md): This feature enables coordination of health checks across multiple LiteLLM proxy pods to avoid duplicate health checks and reduce costs.
- [Auto Sync Anthropic Beta Headers](https://docs.litellm.ai/docs/proxy/sync_anthropic_beta_headers.md): Automatically keep your Anthropic beta headers configuration up to date without restarting your service. This allows you to support new Anthropic beta features across all...
- [User Onboarding Guide](https://docs.litellm.ai/docs/proxy/user_onboarding.md): A step-by-step guide to help admins onboard users to your LiteLLM proxy instance and help users get started with their API key.
- [SearchAPI.io (Google Search)](https://docs.litellm.ai/docs/search/searchapi.md): Get started by creating a free API key via https://www.searchapi.io/.
- [Secret Managers Overview](https://docs.litellm.ai/docs/secret.md): LiteLLM supports reading secrets (eg. OPENAI_API_KEY) and writing secrets (eg. Virtual Keys) from Azure Key Vault, Google Secret Manager, Hashicorp Vault, CyberArk Conjur, and...
- [Codex CLI - Granular Cost Tracking](https://docs.litellm.ai/docs/tutorials/codex_customer_tracking.md): Track Codex CLI usage by customer or tags using LiteLLM proxy. This enables granular cost attribution for billing, budgeting, and analytics.
- [Comparing LLMs on a Test Set using LiteLLM](https://docs.litellm.ai/docs/tutorials/compare_llms_2.md): LiteLLM allows you to use any LLM as a drop in replacement for
- [Customize Prompt Templates on OpenAI-Compatible server](https://docs.litellm.ai/docs/tutorials/model_config_proxy.md): You will learn: How to set a custom prompt template on our OpenAI compatible server.
- [/batchPredictionJobs](https://docs.litellm.ai/docs/vertex_batch_passthrough.md): LiteLLM supports Vertex AI batch prediction jobs through passthrough endpoints, allowing you to create and manage batch jobs directly through the proxy server.