fix(caching): don't trip redis circuit breaker on short timeout bursts

[devin-ai-integration]

TLDR

Problem this solves:

• A burst of Redis timeouts opened the circuit breaker against a healthy Redis
• Event-loop stalls make queued operations time out together, hitting the threshold instantly
• Breaker had no state, transition, or failure-class metrics

How it solves it:

• Timeouts now count separately from hard connectivity failures
• Timeout-only streaks open the breaker only after the threshold count of timeouts persists for REDIS_CIRCUIT_BREAKER_TIMEOUT_MIN_DURATION (default 5s); the duration gate counts timeouts alone, and a hard failure resets the timeout streak, so mixed sequences and stale timeouts cannot open it early
• Hard failures (connection refused, cluster down) still open it at the threshold, unchanged
• Prometheus state gauge labeled by state (counts breakers per state, safe with multiple instances) plus counters for transitions and failures by class

User Flow

Before: a proxy admin running Redis caching sees caching silently stop after a brief latency blip even though Redis is fine

1. The admin runs the proxy with Redis caching and a short client timeout, and sends POST http://localhost:4000/v1/chat/completions traffic
2. A one-off stall makes a handful of Redis reads time out in the same second
3. The proxy log shows "Redis circuit breaker OPENED after 5 consecutive failures" and every cache call is skipped for the next 60 seconds, while redis-cli ping still answers PONG

After: the same blip is ridden out and caching stays on

1. The admin runs the same config and sends the same POST http://localhost:4000/v1/chat/completions traffic
2. The same one-off stall makes the same reads time out
3. All requests return 200, the breaker stays closed, and the very next request reads and writes the cache normally. A real Redis outage (connection refused) still trips the breaker after 5 consecutive failures

Relevant issues

Linear ticket

Resolves LIT-6562

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup: local redis-server on 6379, proxy on port 4000 with cache: {type: redis, socket_timeout: 0.1} and one Anthropic model, real Anthropic API calls. The stall is induced with redis-cli DEBUG SLEEP 6 (Redis stops answering for 6s, then is healthy again), which makes all in-flight cache reads and writes time out together, the same shape as an event-loop stall from the breaker's point of view

Before (3905c9d)

Timeout burst against healthy Redis opens the breaker

1. curl -s http://localhost:4000/chat/completions -H "Authorization: Bearer sk-1234" -d '{"model":"anthropic-haiku-4-5","messages":[{"role":"user","content":"Say OK"}],"max_tokens":5}' returns 200 with a real completion (control)
2. (redis-cli DEBUG SLEEP 6 &) then 8 concurrent identical curls: all return 200 200 200 200 200 200 200 200
3. redis-cli ping right after returns PONG (Redis healthy)
4. Proxy log: 18:52:02 - LiteLLM:WARNING: redis_cache.py:156 - Redis circuit breaker OPENED after 5 consecutive failures — fast-failing Redis calls for 60s, so caching is disabled for 60s against a healthy Redis

After (5481729)

Timeout burst against healthy Redis keeps the breaker closed

1. Same control curl returns control:200
2. Same (redis-cli DEBUG SLEEP 6 &) plus 8 concurrent curls: all 200, worst latency 0.13s
3. redis-cli ping returns PONG; proxy log has 25 Timeout reading from 127.0.0.1:6379 entries and zero circuit breaker OPENED lines, so the burst was seen and ridden out
4. Next request (recovery:200) reads and writes the cache normally

Real outage still fails fast

1. sudo service redis-server stop, then 6 sequential curls: all return 200 (cache skipped, LLM still served)
2. Proxy log: 19:30:47 - LiteLLM:WARNING: redis_cache.py:190 - Redis circuit breaker OPENED after 5 consecutive failures (5 hard connectivity) — fast-failing Redis calls for 60s

Benchmark, one worker, reproduced load profile

1. 8 concurrent completions during the 6s stall on one uvicorn worker: max latency 0.83s, wall clock 0.83s, all 200, breaker closed

Type

🐛 Bug Fix

Caveats (if any)

Low

• The live stall was induced by pausing Redis for 6s rather than pinning the proxy's event loop; both produce the identical observable (a same-second burst of client timeouts against a Redis that answers PONG right after), and the event-loop variant is covered by the regression test that blocks the loop with time.sleep
• Timeout-only streaks now wait REDIS_CIRCUIT_BREAKER_TIMEOUT_MIN_DURATION (default 5s, env-configurable) before opening; a genuinely hung Redis that only times out is evicted ~5s later than before

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/c5971319db694f0db319fbe75f40e09d
Open in Devin Desktop: https://app.devin.ai/desktop/session/c5971319db694f0db319fbe75f40e09d?variant=devin
Requested by: @yassin-berriai

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[devin-ai-integration]

No action taken on #38999 — repo and author check out (devin-ai-integration[bot]), but the PR has no labels, so the required enterprise label is absent and it's out of scope. No GitHub or Linear changes made.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_redis_breaker_timeout_class (5481729) with litellm_internal_staging (a73e770)}

[greptile-apps]

Greptile Summary

The PR separates timeout streaks from hard Redis connectivity failures and adds state and failure-class metrics.

• Timeout-only failures must now meet both a count and duration threshold.
• Hard failures reset timeout streak accounting and continue to open the breaker at their configured threshold.
• State transitions and classified failures are exposed through Prometheus metrics.
• Regression tests cover timeout bursts, persistent timeouts, mixed failures, and metric transitions.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/caching/redis_cache.py	Adds class-specific breaker accounting, a timeout-duration gate, and aggregate state and transition metrics.
litellm/constants.py	Adds the environment-configurable minimum duration for timeout-only breaker activation.
tests/test_litellm/caching/test_redis_cache.py	Adds focused regression coverage for timeout bursts, mixed failure sequences, recovery, and metrics.

_{Reviews (3): Last reviewed commit: "fix(caching): reset the timeout streak o..." | Re-trigger Greptile}

[codecov]

Codecov Report

❌ Patch coverage is 92.20779% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/caching/redis_cache.py	92.10%	6 Missing ⚠️

📢 Thoughts on this report? Let us know!

[devin-ai-integration]

@greptileai

[devin-ai-integration]

Fixed in 5481729: a hard failure now resets the timeout streak, so a later burst must span the duration itself

[devin-ai-integration]

@greptileai