fix(prometheus): bound requested_model label cardinality on client failure paths

[mateo-berri]

TLDR

Problem this solves:

• Every unique junk model name a client sends mints permanent Prometheus series
• requested_model label cardinality grows unbounded, leaking memory until restart
• Agents sending typo'd or hallucinated model names make /metrics huge

How it solves it:

• Failure paths now validate the client model string against the router
• Known names, aliases, team model names, and wildcard-matched names keep their own label
• Everything else collapses into one requested_model="other" series
• SDK Router deployment failure and fallback events, where no proxy router exists to vouch for names the router itself emitted, keep their labels

User Flow

Before: an operator whose agents send typo'd model names watches proxy memory and /metrics grow without bound

1. An agent sends POST https://litellm-domain/v1/chat/completions with "model": "gpt-4o-mimi" (a typo) and gets a 400 back
2. The operator runs curl -s https://litellm-domain/metrics | grep -o 'requested_model="[^"]*"' | sort -u | wc -l and the count went up by one
3. Each new typo'd name repeats this: after a week of agent traffic the count is in the tens of thousands, every scrape ships megabytes, and pod memory climbs until a restart
4. A request with a real model name, POST https://litellm-domain/v1/chat/completions with "model": "gpt-4o-mini", still shows up as requested_model="gpt-4o-mini"

After: the same junk traffic folds into a single label value and the count stays flat

1. An agent sends POST https://litellm-domain/v1/chat/completions with "model": "gpt-4o-mimi" (a typo) and gets a 400 back
2. The operator runs the same grep | sort -u | wc -l and the count did not move: the failure was recorded under requested_model="other"
3. Ten thousand more unique junk names later, the count is still the same, /metrics stays small, and memory stays flat
4. A request with a real model name, POST https://litellm-domain/v1/chat/completions with "model": "gpt-4o-mini", still shows up as requested_model="gpt-4o-mini", and wildcard traffic like "model": "openai/gpt-4o-audio-preview" under an openai/* route keeps its own label too
5. Team-scoped names behave the same: a request against a team model alias or a team wildcard match keeps its own requested_model label

Relevant issues

Linear ticket

Resolves LIT-6611

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Both legs ran against a live proxy with --num_workers 2 (two uvicorn workers confirmed via startup logs and per-pid multiproc db files), a dedicated empty PROMETHEUS_MULTIPROC_DIR, and this config:

model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
  - model_name: openai/*
    litellm_params:
      model: openai/*
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  callbacks: ["prometheus"]
  require_auth_for_metrics_endpoint: false
  prometheus_metrics_config:
    - group: request_tracking
      metrics:
        - litellm_proxy_total_requests_metric
        - litellm_proxy_failed_requests_metric
      include_labels:
        - requested_model
        - end_user
        - hashed_api_key
        - api_key_alias
        - team
        - user
    - group: deployment_health
      metrics:
        - litellm_deployment_total_requests
        - litellm_deployment_success_responses
        - litellm_deployment_failure_responses
      include_labels:
        - requested_model
        - litellm_model_name
        - api_provider
    - group: performance
      metrics:
        - litellm_request_total_latency_metric
        - litellm_llm_api_latency_metric
      include_labels:
        - requested_model
        - model
        - api_provider

general_settings:
  master_key: sk-lit6611-qa-39be45e8cb31cbeef8c90137

The traffic mix hits all three unified endpoints with one unique junk name per request (KEY below is the throwaway master key above):

for n in $(seq 1 50); do curl -s -o /dev/null -w '%{http_code}\n' http://localhost:PORT/v1/chat/completions -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "{\"model\":\"junk-${n}\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}"; done
for n in $(seq 51 100); do curl -s -o /dev/null -w '%{http_code}\n' http://localhost:PORT/v1/responses -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "{\"model\":\"junk-${n}\",\"input\":\"hi\"}"; done
for n in $(seq 101 150); do curl -s -o /dev/null -w '%{http_code}\n' http://localhost:PORT/v1/messages -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' -d "{\"model\":\"junk-${n}\",\"max_tokens\":16,\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}"; done

Each checkpoint below is 10 consecutive curl -sL http://localhost:PORT/metrics scrapes, counting grep -o 'requested_model="[^"]*"' | sort -u | wc -l per scrape; min and max across the 10 were identical at every checkpoint in both legs

Before (4db1653)

Checkpoint	Traffic	Unique labels	HTTP codes
0	1 valid gpt-4o-mini call	1	1x 200
1	150 unique junk names	151	150x 400
2	150 more junk + 2 valid	302	150x 400, 2x 200
3	50 reuses of one junk name	302	50x 400

• Every junk name is a permanent series on all 4 requested_model-labeled counter families (1200 series lines at checkpoint 3), and counts never shrank across any of the 40 scrapes
• Reuse increments instead of minting: junk-1 on litellm_proxy_failed_requests_metric_total went 1.0 to 51.0
• Valid names label as themselves: requested_model="gpt-4o-mini" and requested_model="openai/gpt-4o-mini-2024-07-18" via the wildcard route

After (2adae6b)

Success controls first: real completions on /v1/chat/completions, /v1/responses, and /v1/messages with gpt-4o-mini, plus openai/gpt-4o-mini-2024-07-18 through openai/*, all 200

Checkpoint	Traffic	Unique labels	Label value union
0	4 successes	2	gpt-4o-mini, openai/gpt-4o-mini-2024-07-18
1	150 unique junk names	3	+ other
2	150 more unique junk	3	same 3
3	50 reuses of one junk name	3	same 3

• All junk requests still surfaced as 4xx model-not-found to the client; no 200s, no 5xx
• grep -c qa-typo (the junk name prefix) on every saved snapshot: 0, 0, 0, 0; the raw names never reach /metrics
• The other series absorbs everything: all four counter families read 150.0 at checkpoint 1, 300.0 at checkpoint 2, 350.0 at checkpoint 3, matching the junk request counts exactly
• Known-name labeling unchanged: litellm_deployment_success_responses_total{api_provider="openai",litellm_model_name="gpt-4o-mini",requested_model="gpt-4o-mini"} 3.0 and the wildcard-routed name keeps its full client-sent value
• Scrape payload froze at 14632 bytes from checkpoint 1 on; the before leg's kept growing with every new name

Surprises:

• deployment_health metrics also mint per-name series, doubling the cost
• Concurrent proxies share the auto-created multiproc dir; legs isolated
• Identical after-leg numbers at all three tips reached mid-run

Type

🐛 Bug Fix

Caveats (if any)

Low

• Fallback metrics' requested_model is bounded the same way; junk groups read other
• litellm_llm_api_failed_requests_metric's model label still carries raw strings; left for the generalized series cap
• Strings matching a configured wildcard (any team's included) keep their raw label, so suffix cardinality under wildcards is also left for the generalized cap
• Dashboards keyed on exact junk names (unlikely) now see other
• A configured model literally named other would share its label value with the bucket

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• 2adae6b passes /live-pr-risk

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_lit6611_requested_model_label_cap (2adae6b) with litellm_internal_staging (f3dbd25)}

[greptile-apps]

Greptile Summary

This PR bounds failure-path Prometheus requested_model cardinality while preserving recognized Router names and configured wildcard matches.

• Adds centralized requested-model label classification for proxy, deployment, and fallback failure metrics.
• Preserves SDK Router labels when no proxy Router exists.
• Adds coverage for unknown names, aliases, global and team wildcards, missing proxy routers, and fallback events.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/integrations/prometheus.py	Adds bounded requested-model label handling and now recognizes team wildcard routes, resolving the previously reported classification gap.
tests/enterprise/litellm_enterprise/enterprise_callbacks/test_prometheus_logging_callbacks.py	Supplies a configured Router to existing callback tests so recognized model labels continue to be asserted under the new classification behavior.
tests/test_litellm/integrations/test_prometheus_requested_model_cardinality.py	Adds focused regression coverage for cardinality bounding and preservation of recognized global, team, wildcard, and SDK Router labels.

_{Reviews (4): Last reviewed commit: "fix(prometheus): pass through router-ori..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 95.65217% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/integrations/prometheus.py	95.65%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

@greptileai

[veria-ai]

Low: Wildcard matches retain unbounded label values

An authenticated caller can repeatedly request values such as team-models/<random> and mint a distinct permanent metric series for every failure. This also checks every team's patterns without considering the caller's team, so access to that team route is not required. Emit a finite value such as the configured wildcard pattern instead of the concrete requested model; apply the same treatment to the global wildcard branch above.

[mateo-berri]

Success metrics label wildcard traffic by concrete model; emitting patterns on failure breaks per-model failure dashboards. The authenticated residual is documented for the generalized cap

[veria-ai]

Thanks for the detail. I can't automatically confirm this is safe to dismiss, so I'm leaving the thread open for a maintainer to make the call.

[veria-ai]

PR overview

This pull request updates the Prometheus integration to normalize the requested_model label on client failure metrics, particularly for wildcard model matches.

One gap remains: authenticated callers can use randomized model names matching wildcard routes to create an unbounded number of persistent metric series. This can increase Prometheus storage and resource consumption, although the impact is limited to metrics handling.

Open issues (1)

• Low: Wildcard matches retain unbounded label values — litellm/integrations/prometheus.py:190

Fixed/addressed: 0 · PR risk: 5/10

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 2adae6b. Configure here.}