fix(proxy): share per-model budget counters across replicas through the spend counter cache

[devin-ai-integration]

TLDR

Problem this solves:

• Per-model budgets (model_max_budget) were only enforced on the worker that served the spend
• /key/info and /user/info on any other worker reported current_spend: 0.0
• A capped key kept getting 200s from the other replicas after the cap tripped
• Regression surfaced on v1.99.0-rc.1 on a multi-replica deployment without enable_redis_auth_cache

How it solves it:

• The per-model budget limiter now runs on spend_counter_cache, which always gets Redis
• Enforcement and usage reads go to Redis first when one is attached, in-memory otherwise
• Auth cache semantics are untouched: user_api_key_cache still needs enable_redis_auth_cache

The per-model counters were kept on user_api_key_cache, which only gets a Redis backend when enable_redis_auth_cache: true. Without that opt-in every worker held its own copy of the counter, never pushed increments to Redis, and answered budget checks and /key/info from that local copy. A second layer sat behind the first: even with Redis attached, DualCache reads the in-memory tier first, so a worker that had served part of the traffic enforced and reported its own share rather than the shared total. Both are addressed here, and the per-model spend counters now live next to the other spend counters that were already Redis-backed unconditionally

-model_max_budget_limiter: Final = _PROXY_VirtualKeyModelMaxBudgetLimiter(dual_cache=user_api_key_cache)
+model_max_budget_limiter: Final = _PROXY_VirtualKeyModelMaxBudgetLimiter(dual_cache=spend_counter_cache)

async def _cached_spend(self, spend_key: str) -> float | None:
    redis_cache: Final = self.dual_cache.redis_cache
    if redis_cache is None:
        return await self.dual_cache.async_get_cache(key=spend_key)
    return await redis_cache.async_get_cache(key=spend_key)

build_model_max_budget_usage (behind /key/info and /user/info) does the same with one batched Redis read. This is the same Redis-first read the provider budget limiter in router_strategy/budget_limiter.py already does

User Flow

Before: a key with a $0.01 per-model budget on claude-sonnet-5 is only cut off by the worker that happened to serve its traffic

1. The admin creates the key with POST https://litellm-domain/key/generate and {"model_max_budget": {"claude-sonnet-5": {"budget_limit": 0.01, "time_period": "30d"}}}
2. The developer sends POST https://litellm-domain/v1/chat/completions with that key. The first five return 200 with a chatcmpl-... id, the sixth returns 429 "exceeded budget for model=claude-sonnet-5"
3. The admin opens GET https://litellm-domain/key/info?key=... and, depending on which replica answers, sees model_max_budget_usage.claude-sonnet-5.current_spend as 0.0102 or 0.0
4. The developer's next POST https://litellm-domain/v1/chat/completions lands on another replica and returns 200 with a chatcmpl-... id: the key is over budget and still being served

After: every replica enforces and reports the same shared budget

1. The admin creates the key with POST https://litellm-domain/key/generate and {"model_max_budget": {"claude-sonnet-5": {"budget_limit": 0.01, "time_period": "30d"}}}
2. The developer sends POST https://litellm-domain/v1/chat/completions with that key. The first five return 200 with a chatcmpl-... id, the sixth returns 429 "exceeded budget for model=claude-sonnet-5"
3. The admin opens GET https://litellm-domain/key/info?key=... and every replica reports current_spend: 0.0102 against budget_limit: 0.01
4. The developer's next POST https://litellm-domain/v1/chat/completions, on any replica, returns 429 "exceeded budget for model=claude-sonnet-5"

Relevant issues

Linear ticket

Resolves LIT-6733

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Setup shared by both arms: Postgres, one Redis, and two proxy processes ("pod1" and "pod2") started from the same checkout with PYTHONPATH=<checkout>, the same config, and real Anthropic calls. enable_redis_auth_cache is not set, matching the reported deployment. The three fixture scripts are byte-identical across arms (sha1sum: ace62f2f key, dee7cb34 user, f693818b messages); only the key alias / user id argument names the arm. Which build served a request is pinned with the imported module: hasattr(litellm.proxy.hooks.model_max_budget_limiter, "_current_window_spends") is ABSENT on Before and PRESENT on After

model_list:
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY
general_settings:
  master_key: sk-1234
router_settings:
  redis_host: localhost
  redis_port: 6379
litellm_settings:
  cache: true
  cache_params:
    type: redis
    host: localhost
    port: 6379
    supported_call_types: []

BODY='{"model":"claude-sonnet-5","max_tokens":200,"messages":[{"role":"user","content":"Write a 150 word paragraph about the ocean."}]}'

Before (ba2e5d2)

Key-level budget, /v1/chat/completions

1. curl -s http://localhost:4008/key/generate -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"key_alias":"key-before-ba2e5d2-r2","model_max_budget":{"claude-sonnet-5":{"budget_limit":0.01,"time_period":"30d"}}}'
2. for i in 1 2 3 4 5 6; do curl -s -w "HTTP %{http_code} " http://localhost:4008/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d "$BODY"; done (id or error tail printed)

   HTTP 200 chatcmpl-152ec7d0-f4fb-4fce-82c4-1acaea3eba65
   HTTP 200 chatcmpl-745de0b6-a061-47ec-937a-98a164c345bf
   HTTP 200 chatcmpl-8a3dd235-4601-498d-becc-0e255e55eeb8
   HTTP 200 chatcmpl-0cb7e185-75fc-41f7-ad72-37ba24c835b1
   HTTP 200 chatcmpl-4eabb5a7-97b0-4782-9953-726110c5ed5a
   HTTP 429 before-ba2e5d2-r2, exceeded budget for model=claude-sonnet-5
   

3. curl -s "http://localhost:4008/key/info?key=$KEY" -H "Authorization: Bearer sk-1234" (pod1)

   {"spend": 0.01023, "model_max_budget_usage": {"claude-sonnet-5": {"current_spend": 0.0102, "budget_limit": 0.01, "time_period": "30d"}}}
   

4. curl -s "http://localhost:4009/key/info?key=$KEY" -H "Authorization: Bearer sk-1234" (pod2)

   {"spend": 0.01023, "model_max_budget_usage": {"claude-sonnet-5": {"current_spend": 0.0, "budget_limit": 0.01, "time_period": "30d"}}}
   

5. for i in 1 2; do curl -s -w "HTTP %{http_code} " http://localhost:4009/v1/chat/completions -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -d "$BODY"; done (pod2, cap already exhausted on pod1)

   HTTP 200 chatcmpl-b1463f5b-ea2b-445e-a195-17eb5d337c4a
   HTTP 200 chatcmpl-4e3a8e61-5282-40b9-a4e0-babcf9b959dc
   

6. docker exec lit6733-redis redis-cli --scan --pattern "virtual_key_spend:*"

   (no keys: nothing was pushed to Redis)
   

User-level budget, /v1/chat/completions

1. curl -s http://localhost:4008/user/new -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"user_id":"user-before-ba2e5d2","model_max_budget":{"claude-sonnet-5":{"budget_limit":0.01,"time_period":"30d"}}}' then curl -s http://localhost:4008/key/generate ... -d '{"user_id":"user-before-ba2e5d2"}'
2. 6 x POST http://localhost:4008/v1/chat/completions with that key (same loop as above)

   HTTP 200 chatcmpl-15a483ed-fa0e-4038-a884-69978cac3f92
   HTTP 200 chatcmpl-866c10a5-5270-4deb-99c5-a6eef861a3d4
   HTTP 200 chatcmpl-28308918-cc67-415c-8f63-65b826cdd760
   HTTP 200 chatcmpl-49b0bc64-7882-4cf0-a497-7eb0d98531ce
   HTTP 200 chatcmpl-b4f66fb4-d258-4025-b207-216314f72563
   HTTP 429 M User: user-before-ba2e5d2, exceeded budget for model=claude-sonnet-5
   

3. curl -s "http://localhost:4008/user/info?user_id=user-before-ba2e5d2" -H "Authorization: Bearer sk-1234" (pod1)

   {"spend": 0.008184, "model_max_budget_usage": {"claude-sonnet-5": {"current_spend": 0.0102, "budget_limit": 0.01, "time_period": "30d"}}}
   

4. curl -s "http://localhost:4009/user/info?user_id=user-before-ba2e5d2" -H "Authorization: Bearer sk-1234" (pod2)

   {"spend": 0.008184, "model_max_budget_usage": {"claude-sonnet-5": {"current_spend": 0.0, "budget_limit": 0.01, "time_period": "30d"}}}
   

5. 2 x POST http://localhost:4009/v1/chat/completions (pod2)

   HTTP 200 chatcmpl-d160a393-9d58-416b-a662-55c5ddbf92e3
   HTTP 200 chatcmpl-5a7fd707-2bd5-4868-804b-1ac2114b64dd
   

Key-level budget, /v1/messages

1. curl -s http://localhost:4008/key/generate ... -d '{"key_alias":"msgs-before-ba2e5d2","model_max_budget":{"claude-sonnet-5":{"budget_limit":0.01,"time_period":"30d"}}}'
2. for p in 4008 4008 4008 4008 4008 4008 4009 4009; do curl -s -w "pod :$p HTTP %{http_code} " http://localhost:$p/v1/messages -H "Authorization: Bearer $KEY" -H "anthropic-version: 2023-06-01" -H "Content-Type: application/json" -d "$BODY"; done

   pod :4008 HTTP 200 msg_011Cef8g1f8v9wvcyjWANw16
   pod :4008 HTTP 200 msg_011Cef8gWP2nCqL1TqR6gGkr
   pod :4008 HTTP 200 msg_011Cef8h2CuUY66U4pXmrWp3
   pod :4008 HTTP 200 msg_011Cef8hYHuLTgchxQNHQs1h
   pod :4008 HTTP 200 msg_011Cef8i564gfMqfVUtzHJpJ
   pod :4008 HTTP 429 gs-before-ba2e5d2, exceeded budget for model=claude-sonnet-5
   pod :4009 HTTP 200 msg_011Cef8iuWPDPCG3Gy3yE8an
   pod :4009 HTTP 200 msg_011Cef8jS2vdJGhLPLkKe1mc
   

After (a6f6a79)

Key-level budget, /v1/chat/completions

1. curl -s http://localhost:4006/key/generate -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"key_alias":"key-after-a6f6a79","model_max_budget":{"claude-sonnet-5":{"budget_limit":0.01,"time_period":"30d"}}}'
2. 6 x POST http://localhost:4006/v1/chat/completions (pod1)

   HTTP 200 chatcmpl-e42ccd50-361c-46d8-a527-771cb0e45927
   HTTP 200 chatcmpl-c9843aa5-7be2-4736-af75-45e53a1cad02
   HTTP 200 chatcmpl-ef1ce2fd-6cf3-4459-8d53-7f8c22078dd5
   HTTP 200 chatcmpl-0378f982-82ca-4c8e-b675-9fd9065c3555
   HTTP 200 chatcmpl-63de9b67-e750-4b62-bf7b-148462a31098
   HTTP 429 key-after-a6f6a79, exceeded budget for model=claude-sonnet-5
   

3. curl -s "http://localhost:4006/key/info?key=$KEY" -H "Authorization: Bearer sk-1234" (pod1)

   {"spend": 0.008184, "model_max_budget_usage": {"claude-sonnet-5": {"current_spend": 0.0102, "budget_limit": 0.01, "time_period": "30d"}}}
   

4. curl -s "http://localhost:4007/key/info?key=$KEY" -H "Authorization: Bearer sk-1234" (pod2)

   {"spend": 0.008184, "model_max_budget_usage": {"claude-sonnet-5": {"current_spend": 0.0102, "budget_limit": 0.01, "time_period": "30d"}}}
   

   (spend is the DB total and lags the last spend-log flush by a few seconds; current_spend is the shared per-model counter this PR is about)
5. 2 x POST http://localhost:4007/v1/chat/completions (pod2, cap already exhausted on pod1)

   HTTP 429 key-after-a6f6a79, exceeded budget for model=claude-sonnet-5
   HTTP 429 key-after-a6f6a79, exceeded budget for model=claude-sonnet-5
   

6. docker exec lit6733-redis redis-cli --scan --pattern "virtual_key_spend:*" (one entry per capped key created in this Redis, this run's is the newest)

   virtual_key_spend:5c34aacabb7659ef5eb52d7ce303d7ea6747f921f74c7149872a31cc174592d4:claude-sonnet-5:30d = 0.01023
   

User-level budget, /v1/chat/completions

1. curl -s http://localhost:4006/user/new ... -d '{"user_id":"user-after-a6f6a79","model_max_budget":{"claude-sonnet-5":{"budget_limit":0.01,"time_period":"30d"}}}' then curl -s http://localhost:4006/key/generate ... -d '{"user_id":"user-after-a6f6a79"}'
2. 6 x POST http://localhost:4006/v1/chat/completions (pod1)

   HTTP 200 chatcmpl-39ff6f42-2c42-4ec1-8c43-07c0d24104e2
   HTTP 200 chatcmpl-7735c6bd-1b9b-46d2-9c7d-a9bc0ab7502c
   HTTP 200 chatcmpl-c845e5f5-8d5c-4a3e-a357-b9a637df2313
   HTTP 200 chatcmpl-c11ed41d-9c3c-4d42-b4a0-8bc5269af5b9
   HTTP 200 chatcmpl-2d174dde-a0ec-4278-a567-101ccbb1cf69
   HTTP 429 LM User: user-after-a6f6a79, exceeded budget for model=claude-sonnet-5
   

3. curl -s "http://localhost:4006/user/info?user_id=user-after-a6f6a79" -H "Authorization: Bearer sk-1234" (pod1)

   {"spend": 0.01023, "model_max_budget_usage": {"claude-sonnet-5": {"current_spend": 0.0102, "budget_limit": 0.01, "time_period": "30d"}}}
   

4. curl -s "http://localhost:4007/user/info?user_id=user-after-a6f6a79" -H "Authorization: Bearer sk-1234" (pod2)

   {"spend": 0.01023, "model_max_budget_usage": {"claude-sonnet-5": {"current_spend": 0.0102, "budget_limit": 0.01, "time_period": "30d"}}}
   

5. 2 x POST http://localhost:4007/v1/chat/completions (pod2)

   HTTP 429 LM User: user-after-a6f6a79, exceeded budget for model=claude-sonnet-5
   HTTP 429 LM User: user-after-a6f6a79, exceeded budget for model=claude-sonnet-5
   

6. docker exec lit6733-redis redis-cli --scan --pattern "user_model_spend:user-after-a6f6a79:*"

   user_model_spend:user-after-a6f6a79:claude-sonnet-5:30d = 0.01023
   

Key-level budget, /v1/messages

1. curl -s http://localhost:4006/key/generate ... -d '{"key_alias":"msgs-after-a6f6a79","model_max_budget":{"claude-sonnet-5":{"budget_limit":0.01,"time_period":"30d"}}}'
2. for p in 4006 4006 4006 4006 4006 4006 4007 4007; do curl -s -w "pod :$p HTTP %{http_code} " http://localhost:$p/v1/messages -H "Authorization: Bearer $KEY" -H "anthropic-version: 2023-06-01" -H "Content-Type: application/json" -d "$BODY"; done

   pod :4006 HTTP 200 msg_011CefAo5r5WhJzZX6FPsS7M
   pod :4006 HTTP 200 msg_011CefAobsbzvaw8mcd7VKGx
   pod :4006 HTTP 200 msg_011CefAp7fjzjTELWGEknha4
   pod :4006 HTTP 200 msg_011CefApdpTsPnK6McG6t5mK
   pod :4006 HTTP 200 msg_011CefAqAALYZqJ423fEPC3v
   pod :4006 HTTP 429 sgs-after-a6f6a79, exceeded budget for model=claude-sonnet-5
   pod :4007 HTTP 429 sgs-after-a6f6a79, exceeded budget for model=claude-sonnet-5
   pod :4007 HTTP 429 sgs-after-a6f6a79, exceeded budget for model=claude-sonnet-5
   

Regression tests: test_spend_logged_on_one_replica_is_enforced_and_reported_on_another in tests/proxy_unit_tests/test_unit_test_max_model_budget_limiter.py (two limiters on one fake Redis; fails on the base with DID NOT RAISE BudgetExceededError, and with the Redis-first read removed with 0.25 == 1.25) and test_flag_absent_still_shares_the_model_budget_counters_over_redis in tests/test_litellm/proxy/test_redis_auth_cache_flag.py (fails when the limiter is wired back to user_api_key_cache). make check passes. a6f6a79 only changes the fake Redis in the test to rebind an immutable MappingProxyType instead of mutating a dict, and the mutation check above was rerun on it

Type

🐛 Bug Fix

Caveats (if any)

Low

• Scope: key, user and end-user per-model budgets. Team, org, and provider/deployment budgets are separate counters and unchanged
• Each admission check and /key/info now costs one Redis round trip instead of an in-memory hit when Redis is configured
• Deployments that already set enable_redis_auth_cache: true keep their counters: same Redis, same key names

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/953017d9a83441f6acba4a7739f31fc0
Open in Devin Desktop: https://app.devin.ai/desktop/session/953017d9a83441f6acba4a7739f31fc0?variant=devin
Requested by: @yassin-berriai

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[devin-ai-integration]

PR #39375 (BerriAI/litellm, author devin-ai-integration[bot]) has no labels — the enterprise label is absent, so no risk labeling or Linear routing was performed. No changes made.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_model_budget_shared_counters (a6f6a79) with litellm_internal_staging (a677242)}

[greptile-apps]

Greptile Summary

The PR moves per-model budget accounting to the Redis-backed spend counter cache so enforcement and usage reporting share totals across proxy replicas.

• Reads shared per-model spend directly from Redis when available, with an in-memory fallback.
• Wires the model budget limiter to spend_counter_cache.
• Adds cross-replica enforcement, reporting, and cache-configuration regression coverage.
• Updates the fake Redis helper to rebind an immutable MappingProxyType store instead of mutating a dictionary in place.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/proxy/hooks/model_max_budget_limiter.py	Adds Redis-first single and batched spend reads while preserving the existing in-memory fallback.
litellm/proxy/proxy_server.py	Moves the per-model budget limiter from the optional Redis auth cache to the always-attached spend counter cache.
tests/proxy_unit_tests/test_unit_test_max_model_budget_limiter.py	Adds cross-replica budget regression coverage and now keeps the fake Redis store immutable through MappingProxyType rebinding.
tests/test_litellm/proxy/test_redis_auth_cache_flag.py	Verifies per-model budget counters remain Redis-backed without enabling the Redis authentication cache.

_{Reviews (2): Last reviewed commit: "test(proxy): keep the shared fake Redis ..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai please re-review at a6f6a79, the only change since f74437d is the immutable fake Redis store in the test