Add CodSpeed continuous performance benchmarks

[codspeed]

Summary

This PR integrates CodSpeed for continuous performance monitoring of litellm's core hot paths.

Changes

• Benchmark suite (tests/benchmarks/test_benchmarks.py): 16 benchmarks using pytest-codspeed covering the most performance-critical functions:
  • Token counting: simple messages, multi-turn conversations, long content, tool definitions, and raw text
  • Model info lookup: OpenAI, Anthropic, and provider-prefixed models
  • Provider resolution: OpenAI, Anthropic, prefixed, and Azure models
  • Cost calculation: per-token cost for OpenAI and Anthropic
  • Model cost key resolution: exact match and case-insensitive fallback
• CI workflow (.github/workflows/codspeed.yml): GitHub Actions workflow triggered on pushes to main, pull requests, and manual dispatch. Uses OIDC authentication and CPU simulation mode.
• README badge: CodSpeed performance badge added to the badge section.

Next steps

• Once merged, CodSpeed will begin collecting baseline performance data on main.
• Future PRs will automatically receive performance regression reports as check results.
• Additional benchmarks can be added to tests/benchmarks/ following the same @pytest.mark.benchmark pattern.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
litellm	Ready	Preview, Comment	Mar 15, 2026 1:32am

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codspeed]

Congrats! CodSpeed is installed 🎉

🆕 16 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

• test_cost_per_token_openai: 543.3 µs
• test_cost_per_token_anthropic: 543.5 µs
• test_get_model_info_openai: 86.5 µs
• test_get_llm_provider_openai: 143.2 µs
• test_get_llm_provider_anthropic: 147.7 µs
• test_token_counter_with_tools: 416.6 µs
• test_get_llm_provider_azure: 149.4 µs
• test_token_counter_long_content: 1.7 ms
• test_get_model_info_with_provider: 85.4 µs
• test_token_counter_simple_message: 239.5 µs
• test_token_counter_raw_text: 187.9 µs
• test_token_counter_multi_turn: 571.6 µs
• test_get_llm_provider_with_prefix: 142.8 µs
• test_get_model_cost_key_exact_match: 85.1 µs
• test_get_model_info_anthropic: 84.4 µs
• test_get_model_cost_key_case_insensitive: 87.6 µs

---

[greptile-apps]

Greptile Summary

This PR integrates CodSpeed for continuous performance monitoring of litellm's hot paths by adding a benchmark suite, a GitHub Actions workflow, and a README badge.

Key changes:

• tests/benchmarks/test_benchmarks.py: 16 benchmarks using pytest-codspeed covering token_counter, get_model_info, get_llm_provider, cost_per_token, and _get_model_cost_key. All benchmarked functions operate on local data with no network calls, which is consistent with the repository's policy for this test directory.
• .github/workflows/codspeed.yml: Well-structured GitHub Actions workflow using OIDC authentication and simulation mode, matching CodSpeed's recommended configuration. tiktoken (required for token counting) is correctly pulled in as a transitive dependency via pip install -e ".".
• README.md: Adds a CodSpeed badge, but the badge image URL (https://codspeed.io/badge.json) is missing the repository path (BerriAI/litellm) and will not display project-specific data.
• tests/benchmarks/__init__.py: Empty package init file — no issues.

Confidence Score: 4/5

• Safe to merge with one minor fix recommended for the badge URL.
• The benchmarks are well-structured, make no network calls, and correctly cover litellm's hot paths. The CI workflow is properly configured with OIDC auth. The only concrete issue is the non-project-specific badge image URL in the README, and the stylistic concern of accessing a private function via litellm.utils._get_model_cost_key rather than a direct import.
• README.md — badge image URL needs the project path (BerriAI/litellm) added to display correctly.

Important Files Changed

Filename	Overview
.github/workflows/codspeed.yml	New GitHub Actions workflow that installs litellm and pytest-codspeed, then runs benchmarks via the CodSpeedHQ/action@v4 in simulation mode with OIDC authentication. Configuration is correct and follows CodSpeed's recommended setup.
README.md	Adds a CodSpeed performance badge. The badge image URL is missing the project-specific path (BerriAI/litellm) and will not display the correct project stats.
tests/benchmarks/init.py	Empty init file that makes the benchmarks directory a Python package. No issues.
tests/benchmarks/test_benchmarks.py	16 benchmarks covering token counting, model info lookup, provider resolution, cost calculation, and model cost key resolution. All tested functions are local (no network calls). Two benchmarks access a private function via litellm.utils._get_model_cost_key instead of a direct import, which is fragile.

Sequence Diagram

sequenceDiagram
    participant GH as GitHub Actions
    participant CS as CodSpeed Action
    participant PT as pytest
    participant LM as litellm (local)

    GH->>CS: Trigger workflow (push/PR/dispatch)
    CS->>PT: Run pytest tests/benchmarks/ --codspeed
    PT->>LM: token_counter(model, messages/text)
    PT->>LM: get_model_info(model)
    PT->>LM: get_llm_provider(model)
    PT->>LM: cost_per_token(model, tokens)
    PT->>LM: utils._get_model_cost_key(key)
    LM-->>PT: Local results (no network calls)
    PT-->>CS: Benchmark measurements
    CS->>CS: Authenticate via OIDC
    CS-->>GH: Upload results to CodSpeed
    GH-->>GH: Post performance report on PR

Loading

_{Last reviewed commit: 03b0083}

[greptile-apps]

Badge URL missing project path

The badge image URL points to a generic https://codspeed.io/badge.json endpoint which does not include the project name. This will either fail to load or show a generic/empty badge rather than litellm's specific performance stats.

The standard CodSpeed badge format includes the owner and repository in the path:

Suggested change

	<img src="https://img.shields.io/endpoint?url=https://codspeed.io/badge.json" alt="CodSpeed"/>
	<img src="https://img.shields.io/endpoint?url=https://codspeed.io/BerriAI/litellm/badge.json" alt="CodSpeed"/>

[greptile-apps]

Benchmarking a private implementation detail

litellm.utils._get_model_cost_key is a private function (underscore-prefixed). Benchmarking it directly via module attribute access creates a fragile coupling — if the function is renamed, moved, or its internal contract changes, these benchmarks will break silently rather than helping catch regressions.

Consider importing it explicitly at the top of the file to make the dependency clear:

from litellm.utils import _get_model_cost_key

and then calling:

@pytest.mark.benchmark
def test_get_model_cost_key_exact_match():
    _get_model_cost_key("gpt-4o")

@pytest.mark.benchmark
def test_get_model_cost_key_case_insensitive():
    _get_model_cost_key("GPT-4o")

This also surfaces an import error immediately at collection time if the function moves, rather than at benchmark runtime.