feat(complexity-router): add classification_mode to skip classifier on continuation turns

[tin-berri]

TLDR

Problem this solves:

• Agent tool loops classify every request, paying classifier latency each turn
• Continuations can classify differently, switching models mid-loop
• Benchmark: 412 classifier calls where 63 would do (report below)

How it solves it:

• New classification_mode setting: every_request (default, today's behavior) or user_turn
• user_turn classifies only new human asks; continuations reuse the session's held decision
• Skipped turns log a new routing cause, user_turn_continuation

An agentic benchmark comparing the two modes measured 85% fewer classifier calls (63 vs 412), zero mid-loop model switches (vs 18), and 0.84s vs 1.40s median continuation latency, with solve rate a wash

User Flow

Before: an operator routes a coding agent through the complexity router and every tool-result continuation pays classifier latency, sometimes landing on a different model mid-loop

1. The operator points Claude Code at the router model and the user asks a hard question
2. POST /v1/messages classifies COMPLEX and the strong model answers with a tool call
3. The agent sends the tool result back; the request waits on the classifier again before any model sees it
4. That continuation can classify differently, so the loop silently hops models mid-task

After: the same session classifies once per human ask and every continuation goes straight to the already-chosen model

1. The operator adds classification_mode: user_turn under complexity_router_config and restarts the proxy
2. The operator points Claude Code at the router model and the user asks a hard question
3. POST /v1/messages classifies COMPLEX and the strong model answers with a tool call
4. The agent sends the tool result back; it goes straight to the same strong model with no classifier wait
5. The user's next question in the same session classifies fresh, so a trivial follow-up drops back to the cheap tier

Relevant issues

• Open PR feat(complexity-router): add user_turn classification mode that holds one target per tool loop #38773 covers similar ground; this is an independent implementation built from scratch as requested
• The report driving the feature is summarized in the TLDR: 63 vs 412 classifier calls, 0 vs 18 mid-loop switches, 0.84s vs 1.40s continuation latency

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Shared setup for both sides. Real OpenAI models (gpt-4o-mini for SIMPLE/MEDIUM, gpt-4o for COMPLEX/REASONING) behind a live gateway, LLM classifier on gpt-4o-mini. The identical config boots on both shas because the router config tolerates unknown keys, so at the merge base the classification_mode key is simply ignored

model_list:
  - model_name: tin-classmode-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        classification_mode: user_turn
        classifier_type: llm
        classifier_llm_config:
          model: gw-mini
          timeout_ms: 15000
        tiers:
          SIMPLE: gw-mini
          MEDIUM: gw-mini
          COMPLEX: gw-4o
          REASONING: gw-4o
  - model_name: gw-mini
    litellm_params: {model: openai/gpt-4o-mini, api_base: os.environ/GW_BASE, api_key: os.environ/GW_KEY}
  - model_name: gw-4o
    litellm_params: {model: openai/gpt-4o, api_base: os.environ/GW_BASE, api_key: os.environ/GW_KEY}
litellm_settings:
  drop_params: true

Launched with python litellm/proxy/proxy_cli.py --config config.yaml --port <port> --detailed_debug from each checkout. The discriminator is the proxy debug log line ComplexityRouter: routing decision cause=...: cause=llm_classifier means a paid classifier call ran for that request, cause=user_turn_continuation means it was skipped. The source line number in each log line also pins which build produced it (complexity_router.py:2641 at the merge base, :2685/:2406 at the evidence sha). All evidence below was captured at 7d51d4d; the final sha differs from it only by a whitespace-only ruff format amend to one file (+1/-2 lines), which shifts those two tip line numbers to :2684/:2405

Before (d619b1c)

/v1/chat/completions tool loop: every request classifies

1. Ran a 3-request tool-loop session (human ask with a read_file tool, then two tool-result continuations, same metadata.session_id) against the merge-base proxy
2. All three requests routed openai/gpt-4o (x-litellm-model-name header), and the log shows three classifier calls, including both continuations:

18:32:55 ... complexity_router.py:2641 - routing decision cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:32:55 ... complexity_router.py:2641 - routing decision cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:32:55 ... complexity_router.py:2641 - routing decision cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o

New human ask in the same session

1. Ran a session with a COMPLEX ask, a tool continuation, then a new trivial ask ("thanks! what is 2+2?")
2. Headers: gpt-4o, gpt-4o, gpt-4o-mini; the middle continuation also paid a classifier call:

18:36:24 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:36:24 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:36:24 ... cause=llm_classifier, tier=SIMPLE, ... routed_model=gw-mini

/v1/messages tool_result continuation

1. Sent an Anthropic-format human ask plus a pure tool_result continuation (newest message is role=user carrying only a tool_result block), same session
2. Both requests classified:

18:36:29 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:36:29 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o

/v1/responses function_call_output continuation

1. Sent a Responses-API human ask plus a function_call/function_call_output continuation, same session
2. Both requests classified:

18:36:30 ... cause=llm_classifier, tier=REASONING, ... routed_model=gw-4o
18:36:30 ... cause=llm_classifier, tier=REASONING, ... routed_model=gw-4o

Complex session holds tier on a trivial tool output

1. Sent a COMPLEX ask, then a continuation whose newest content is a trivial tool output (a timestamp string)
2. The continuation stayed on gpt-4o here, but only because the classifier happened to score the whole transcript COMPLEX again; it still paid the classifier call (this is the mid-loop-switch risk the report measured at 18 switches):

18:38:10 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:38:10 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o

Real Claude Code client session

1. Pointed the real client at the proxy: ANTHROPIC_BASE_URL=http://127.0.0.1:4602 ANTHROPIC_AUTH_TOKEN=sk-... ANTHROPIC_MODEL=tin-classmode-router claude -p "..." plus claude -p --continue "..." follow-ups
2. Two trivial asks, then a complex ask that used a tool; every request classified, including the intra-turn tool continuation at 18:42:27:

18:38:38 ... cause=llm_classifier, tier=SIMPLE, ... routed_model=gw-mini
18:41:42 ... cause=llm_classifier, tier=SIMPLE, ... routed_model=gw-mini
18:42:21 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:42:27 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o

Every-request behavior in default mode

1. At the merge base there is no mode to set; the runs above are the default and only behavior: every request pays a classifier call

After (7d51d4d)

/v1/chat/completions tool loop: one classify, continuations skip

1. Ran the identical 3-request tool-loop session against the tip proxy with classification_mode: user_turn
2. All three requests routed openai/gpt-4o with the same finish reasons as before, and the log shows one classifier call plus two skips:

18:27:28 ... complexity_router.py:2685 - routing decision cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:27:29 ... complexity_router.py:2406 - routing decision cause=user_turn_continuation, routed_model=gw-4o
18:27:32 ... complexity_router.py:2406 - routing decision cause=user_turn_continuation, routed_model=gw-4o

New human ask in the same session

1. Ran the identical COMPLEX ask, tool continuation, then new trivial ask session
2. Headers: gpt-4o, gpt-4o, gpt-4o-mini; the continuation skipped the classifier and the new ask re-classified down to the cheap tier:

18:27:54 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:27:56 ... cause=user_turn_continuation, routed_model=gw-4o
18:27:58 ... cause=llm_classifier, tier=SIMPLE, ... routed_model=gw-mini

/v1/messages tool_result continuation

1. Sent the identical Anthropic-format pair
2. The ask classified, the tool_result continuation skipped and held the model:

18:28:15 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:28:17 ... cause=user_turn_continuation, routed_model=gw-4o

/v1/responses function_call_output continuation

1. Sent the identical Responses-API pair
2. Same result on this surface:

18:28:24 ... cause=llm_classifier, tier=REASONING, ... routed_model=gw-4o
18:28:28 ... cause=user_turn_continuation, routed_model=gw-4o

Complex session holds tier on a trivial tool output

1. Sent the identical bait pair: COMPLEX ask, then a continuation whose newest content is a trivial timestamp
2. The continuation held gpt-4o by design, with no classifier call to get it wrong:

18:29:05 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:29:05 ... cause=user_turn_continuation, routed_model=gw-4o

Real Claude Code client session

1. Pointed the real client at the tip proxy the same way (ANTHROPIC_BASE_URL=http://127.0.0.1:4600 ... claude -p / claude -p --continue)
2. A simple ask that used the Read tool, then a complex design ask that used it again: each human turn classified once, each intra-turn tool continuation skipped and held its model:

18:12:59 ... cause=llm_classifier, tier=SIMPLE, ... routed_model=gw-mini
18:13:04 ... cause=user_turn_continuation, routed_model=gw-mini
18:13:19 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:13:30 ... cause=user_turn_continuation, routed_model=gw-4o

3. Driving the real client caught an edge synthesized curls could not: Claude Code appends a trailing system-role reminder after the newest user message, so the continuation check compares against the newest non-system message (regression test included)

Every-request behavior in default mode

1. Ran the same 3-request tool-loop session against the tip built proxy with classification_mode omitted from the config
2. All three requests classified, matching the merge base exactly, so existing deployments see no change:

18:30:42 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:30:43 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o
18:30:43 ... cause=llm_classifier, tier=COMPLEX, ... routed_model=gw-4o

Type

🆕 New Feature

Caveats (if any)

Low

• Skipped classifier turns record no savings accounting; classification spend was never counted either, out of scope
• Held-decision TTL semantics come from the existing session-affinity pin unchanged
• 14 pre-existing failures in the mapped test file (all semantic_router-dependent keyword tests, module missing locally); proven identical on the unfixed merge base
• OpenAPI snapshot unchanged is correct: it does not carry these router-config schemas; schema.d.ts regenerated
• The osv-scan CI red (GHSA-h7x2-h6g9-p789, mlflow 3.15.0 in uv.lock) is repo-wide and unrelated: this branch touches no lockfile, every other open PR fails on the identical advisory, and the scanner reports 0 of the finding as fixable since no patched release exists

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes pre-routing behavior for agentic multi-turn traffic when operators enable user_turn; mis-detection of “new ask” vs continuation could skip classification or switch models, though default every_request preserves current behavior.

Overview
Adds classification_mode on the complexity router (every_request default, or user_turn). In user_turn, only requests whose newest non-system turn is a new human ask run classification; agent loop continuations (tool results, chat tool roles, etc.) replay the session’s cached routing pin when session_id is present, logging cause user_turn_continuation. New human asks still re-classify (unlike full session_affinity), and if both flags are on, session affinity wins and pins every turn.

Implementation wires user_turn into the existing tier/deployment pin machinery (_uses_tier_pin / _uses_deployment_pin), gates pin replay with _newest_turn_is_human_ask (ignores trailing system reminders), and is disabled when plugins are configured. Dashboard and OpenAPI types expose the setting and the new routing cause.

^{Reviewed by Cursor Bugbot for commit 160adc5. Bugbot is set up for automated code reviews on this repo. Configure here.}

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 4.5/5

The implementation is focused and well covered. It preserves the default every_request behavior, adds strict config validation, and correctly reuses the existing session pin only for continuation turns in user_turn mode. The new-turn detection handles chat tool messages, Anthropic tool_result payloads, hybrid ask/tool-result turns, trailing system reminders, missing session IDs, plugin suppression, repinning after a new ask, and the interaction with session_affinity. The tests directly verify classifier call counts, model stability, routing causes, and tier changes, while the generated schema and dashboard label are updated consistently.

I’m holding back the final 0.5 only because the PR’s CI status is still pending; the code review itself shows no blocking correctness issue.

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_classification_mode (160adc5) with litellm_internal_staging (66295e7)}

[greptile-apps]

Greptile Summary

The PR adds an opt-in user_turn classification mode that reuses a session’s held routing decision for agent-loop continuations while reclassifying new human asks.

• Extends complexity-router configuration and routing-decision types.
• Adds continuation detection and session-pin replay behavior.
• Adds regression tests for supported message shapes, default behavior, plugins, and session affinity.
• Exposes the new routing cause in dashboard types and log presentation.

Confidence Score: 3/5

The PR does not appear safe to merge until continuation routing is isolated from concurrent turns and stale decisions from non-pinnable new asks.

The current session-level pin has no turn identity and is overwritten unconditionally, while fallback, housekeeping, and plan-mode asks can leave an earlier pin intact; continuations can consequently route to a model chosen for a different turn.

Files Needing Attention: litellm/router_strategy/complexity_router/complexity_router.py

Important Files Changed

Filename	Overview
litellm/router_strategy/complexity_router/complexity_router.py	Implements continuation detection and pin replay, but the session-scoped state still permits cross-turn overwrites and stale-pin reuse.
litellm/router_strategy/complexity_router/config.py	Adds the validated, backward-compatible classification_mode configuration with every_request as the default.
litellm/types/utils.py	Adds the user_turn_continuation routing-decision cause.
tests/test_litellm/router_strategy/test_complexity_router.py	Covers sequential continuation behavior and supported message shapes, but does not resolve the outstanding shared-state failures.
ui/litellm-dashboard/src/components/view_logs/LogDetailsDrawer/RoutingDecisionCard.tsx	Adds a dashboard label for continuation decisions that skip classification.
ui/litellm-dashboard/src/lib/http/schema.d.ts	Synchronizes generated dashboard types with the new configuration value and routing cause.

_{Reviews (3): Last reviewed commit: "feat(complexity-router): add classificat..." | Re-trigger Greptile}

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 7d51d4d. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[tin-berri]

Amended a whitespace-only ruff format fix. @greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Score: 5/5.

The amended Ruff formatting issue is resolved. The change remains focused, preserves the every_request default, and the added coverage exercises the key user_turn behaviors: tool-loop continuations, new asks, trailing system reminders, Messages/Responses shapes, missing session IDs, plugins, and interaction with session_affinity. The schema and dashboard routing-cause updates are also consistent with the implementation, so I have no remaining review concerns.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit e3d275c. Configure here.}

[tin-berri]

@greptile can you review?

[greptile-apps]

Score: 4/5

The implementation is well-scoped and addresses the stated problem: every_request remains the default, while user_turn reuses the held routing decision only for continuation traffic and reclassifies new human asks. The continuation detection covers chat-completions tool turns, Anthropic tool_result turns, Responses API outputs, hybrid asks, and trailing system reminders. It also preserves plugin and session_affinity precedence, adds the new routing cause plus dashboard/schema updates, and includes meaningful regression coverage. The available E2E status is passing.

I’m keeping this at 4 rather than 5 because this changes pre-routing behavior and pin lifecycle across multiple request surfaces, so it carries medium integration risk; the targeted tests and preserved default behavior substantially mitigate that risk. I found no blocking correctness issue in the reviewed diff.

[veria-ai]

PR overview

All previously flagged issues have been addressed. No open security concerns remain on this pull request.

Security review

No open security issues remain on this pull request.

Fixed/addressed: 1 · PR risk: 0/10

[tin-berri]

Rebased onto latest staging to pick up the upstream savings-test fix; code is unchanged. @greptileai can you reply here with a score and why that score?

bugbot run

[greptile-apps]

Score: 3/5

The rebase does not change the reviewed implementation, and the available E2E check is passing. The feature is well-scoped, preserves the every_request default, and has strong coverage for tool-loop continuations, new asks, trailing system reminders, provider-specific message shapes, missing session IDs, plugins, and session_affinity precedence.

I’m keeping this at 3 rather than 4–5 because the opt-in replay decision is still derived entirely from caller-supplied message roles in litellm/router_strategy/complexity_router/complexity_router.py:2346. A caller can submit a fresh human message followed by forged assistant/tool entries, causing it to be treated as continuation traffic and replaying the existing session pin instead of classifying the fresh ask. The server needs a trusted session-side continuity check before replaying a pin. The pin lifecycle also remains vulnerable to overlapping turns and stale pins when a newly classified ask does not produce a pinnable decision. These are correctness and cost-policy concerns in the new mode, so I would not rate the PR as fully safe to merge without addressing them.

The reply submission tool was permission-gated and could not post this comment.

[tin-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 160adc5. Configure here.}

[tin-berri]

Both findings refuted: forged roles answered on veria's thread; unpinnable causes are one-call transients by design (_decision_is_pinnable, complexity_router.py:719), plan continuations re-apply the floor in replay

@greptileai can you reply here with a score and why that score?

[greptile-apps]

Score: 5/5.

Both findings are resolved/refuted on re-review. The role-detection concern is addressed in the existing discussion on Veria’s thread, and the unpinnable-decision path is intentional: _decision_is_pinnable (complexity_router.py:719) makes those decisions one-call transients, so they are not incorrectly replayed. Plan continuations also re-apply the plan floor during pin replay. With those invariants, user_turn only replays a valid held decision, while the default every_request behavior remains unchanged. The added tests cover the classification gate, repinning across new asks, tool-result shapes, trailing system reminders, plugin suppression, and session-affinity precedence. I have no remaining blocking concerns.

[tin-berri]

@greptileai can you review pls with this context: Both findings refuted: forged roles answered on veria's thread; unpinnable causes are one-call transients by design (_decision_is_pinnable, complexity_router.py:719), plan continuations re-apply the floor in replay

https://github.com/greptileai can you reply here with a score and why that score?

@greptile-apps
greptile-apps Bot
commented
44 minutes ago
Contributor
Score: 5/5.

Both findings are resolved/refuted on re-review. The role-detection concern is addressed in the existing discussion on Veria’s thread, and the unpinnable-decision path is intentional: _decision_is_pinnable (complexity_router.py:719) makes those decisions one-call transients, so they are not incorrectly replayed. Plan continuations also re-apply the plan floor during pin replay. With those invariants, user_turn only replays a valid held decision, while the default every_request behavior remains unchanged. The added tests cover the classification gate, repinning across new asks, tool-result shapes, trailing system reminders, plugin suppression, and session-affinity precedence. I have no remaining blocking concerns.

[greptile-apps]

Score: 5/5.

The follow-up context resolves both prior concerns. Forged-role handling is covered in Veria’s existing thread, and _decision_is_pinnable at complexity_router.py:719 intentionally keeps unpinnable outcomes as one-call transients, so they are not replayed on continuations. Plan continuations also re-apply the plan floor during replay. As a result, user_turn replays only valid held decisions, while the default every_request behavior remains unchanged. The tests cover the classification gate, new-ask repinning, tool-result shapes, trailing system reminders, plugin suppression, and session-affinity precedence. I have no remaining blocking concerns.

[tin-berri]

@greptileai is this score still a 3/5 because of these issues:
Greptile Summary
The PR adds an opt-in user_turn classification mode that reuses a session’s held routing decision for agent-loop continuations while reclassifying new human asks.

Extends complexity-router configuration and routing-decision types.
Adds continuation detection and session-pin replay behavior.
Adds regression tests for supported message shapes, default behavior, plugins, and session affinity.
Exposes the new routing cause in dashboard types and log presentation.
Confidence Score: 3/5
The PR does not appear safe to merge until continuation routing is isolated from concurrent turns and stale decisions from non-pinnable new asks.

The current session-level pin has no turn identity and is overwritten unconditionally, while fallback, housekeeping, and plan-mode asks can leave an earlier pin intact; continuations can consequently route to a model chosen for a different turn.

[greptile-apps]

No—the 3/5 assessment is stale. I would keep this at 5/5.

The caller-controlled role concern was addressed in the existing Veria thread and does not create an authorization boundary for this routing decision. _decision_is_pinnable at litellm/router_strategy/complexity_router/complexity_router.py:719 intentionally keeps fallback, housekeeping, and other unpinnable outcomes as one-call transients, so they cannot become stale continuation pins. Plan-mode continuations also re-apply the plan floor during replay.

With those invariants, user_turn replays only a valid held decision, while every_request remains unchanged by default. The targeted coverage and passing E2E check support the 5/5 assessment; I have no remaining blocking concern from the cited findings.