feat(shadow_eval): scope a job to model groups, ANDed with its key, team, and user targets

[tin-berri]

TLDR

Problem this solves:

• Shadow eval scopes only by key, team, or user, never by model
• "Shadow this user's traffic on model X" was not expressible
• A models field in the start body was silently ignored

How it solves it:

• New models list on the job, ANDed with every target
• Matched on the requested model group, alias and target are one name
• A name nobody on the proxy can ask for is a 400 at start
• Forward jobs only: a reverse job already samples its own router's traffic
• Dashboard start form gains an "Only on models" picker

User Flow

Before: an admin wants to shadow one user's haiku-real traffic, but the job samples everything the user sends

1. They send POST https://litellm-domain/auto_router/shadow_eval/start with {"user_ids": ["ms-user-1"], "models": ["haiku-real"], "router_name": "auto-router-cheap", ...}
2. They get 201 and a job, but the response carries no models field; the filter was dropped
3. The user sends requests on haiku-real and sonnet-real from two of their keys
4. GET https://litellm-domain/auto_router/shadow_eval/{job_id} shows all four judged, with by_current_model listing both models

After: the same job samples only the user's haiku-real requests, across every key they own

1. They send the same POST https://litellm-domain/auto_router/shadow_eval/start body
2. They get 201 and the job echoes "models": ["haiku-real"]
3. The user sends requests on haiku-real, sonnet-real, and the alias fast-haiku from two keys, over /chat/completions, /v1/messages and /v1/responses
4. GET https://litellm-domain/auto_router/shadow_eval/{job_id} shows exactly the haiku-real and alias requests judged; by_current_model lists one model and the funnel counts nothing for the sonnet-real requests
5. A typo in models fails at start with 400 models not served by this proxy: 'haiku-reel'
6. On https://litellm-domain/ui/?page=cost-optimization the start form has an "Only on models" picker and the job headline reads "Shadowing 100% of msuser1@example.com traffic on haiku-real via auto-router-cheap"

Relevant issues

• Adds a per-job model-group scope to shadow eval, composable with the existing key, team, and user targets
• A job with no models behaves exactly as before, and every existing job row reads as unscoped
• Model-only jobs (no identity target) stay rejected, and so does a scope on a reverse job; one active job per target and direction is unchanged

Linear ticket

Resolves LIT-7013

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Rig: proxy from this worktree on :4640, real Postgres, upstream models served through the sandbox gateway (real Anthropic calls, real spend). ms-user-1 owns two keys, ms-key-a and ms-key-b, both allowed haiku-real, sonnet-real and auto-router-cheap. router_settings.model_group_alias maps fast-haiku to haiku-real. The judge is judge-sonnet45 (claude-sonnet-4-5), which serves no tier of the router under eval. Every traffic round waited out the ~10s jobs cache before sending

Before (d23bec8)

A model filter on the start body is accepted and dropped

1. curl -X POST :4640/auto_router/shadow_eval/start -d '{"user_ids":["ms-user-1"],"router_name":"auto-router-cheap","shadow_percentage":100,"judge_model":"judge-sonnet45","max_budget":1.0,"models":["haiku-real"],"model_names":["haiku-real"],"model_group":"haiku-real"}'
2. HTTP 201, response has targets: [{"target_type":"user","target_id":"ms-user-1",...}], router_names: ["auto-router-cheap"] and no models key anywhere

The job samples every model the user touches

1. Four requests to /chat/completions: haiku-real and sonnet-real from ms-key-a, then the same pair from ms-key-b
2. curl :4640/auto_router/shadow_eval/1ad0464e-... after the pipeline drained:

   judged_count 4 error_count 0 last_error None
   by_current_model: {"group": "openai/anthropic/claude-sonnet-5", "turn_count": 2, ...}
   by_current_model: {"group": "openai/anthropic/claude-haiku-4-5", "turn_count": 2, ...}
   target user ms-user-1 attempts 4 spend 0.010255
   

Model-only scope is not a job

1. curl -X POST :4640/auto_router/shadow_eval/start -d '{"models":["haiku-real"],"router_name":"auto-router-cheap",...}'
2. HTTP 422 at least one target is required: pass api_key_ids, team_ids, or user_ids

After (8000cfb)

A model filter on the start body is stored and echoed

1. curl -X POST :4640/auto_router/shadow_eval/start -d '{"user_ids":["ms-user-1"],"models":["haiku-real"],"router_name":"auto-router-cheap","shadow_percentage":100,"judge_model":"judge-sonnet45","max_budget":1.0}'
2. HTTP 201, response carries "models":["haiku-real"] beside "router_names":["auto-router-cheap"]

The job samples only the user's requests on that model, alias included, across keys and surfaces

1. Six requests: /chat/completions on haiku-real and sonnet-real from ms-key-a, the same pair from ms-key-b, /v1/messages on the alias fast-haiku from ms-key-a, /v1/responses on sonnet-real from ms-key-b
2. curl :4640/auto_router/shadow_eval/c8c59a6d-... after the pipeline drained:

   models scope: ['haiku-real'] | judged 3 errors 0 last_error None
     by_current_model openai/anthropic/claude-haiku-4-5 turns 3
     funnel not_sampled 0
   

   Three judged: the two haiku-real requests (one per key) and the alias request. The three sonnet-real requests were neither sampled nor counted as skipped, because they are outside the job's traffic

A name the proxy does not serve fails at start

1. curl -X POST :4640/auto_router/shadow_eval/start -d '{"user_ids":["ms-user-1"],"models":["haiku-real","haiku-reel"],...}'
2. HTTP 400 {"detail":"models not served by this proxy: 'haiku-reel'"}

Model-only scope is still not a job

1. curl -X POST :4640/auto_router/shadow_eval/start -d '{"models":["haiku-real"],"router_name":"auto-router-cheap",...}'
2. HTTP 422 at least one target is required: pass api_key_ids, team_ids, or user_ids

A scope on a reverse job is rejected

1. curl -X POST :4640/auto_router/shadow_eval/start -d '{"user_ids":["ms-user-1"],"models":["haiku-real"],"router_name":"auto-router-cheap","direction":"reverse","baseline_model":"sonnet-real",...}'
2. HTTP 422 models is only meaningful for a forward job; a reverse job samples its own router's traffic

Dashboard

1. Open http://localhost:4000/ui/?page=cost-optimization and scroll to "Start a shadow eval"
2. Pick a user under "Users to shadow", then pick one model under "Only on models" (the picker lists the plain model groups configured on the proxy, placeholder "Every model the targets use"; it disappears when the direction is switched to the regression check)
3. Pick an auto-router and a judge, click "Start shadow eval"
4. The new job card's headline reads "Shadowing N% of traffic on via "; a job started with no models picked reads as before

Type

🆕 New Feature

Caveats (if any)

Medium

• One active job per target and direction is unchanged, so two model-scoped jobs on the same user cannot run at once; stop the first
• Alias resolution reads the router's model_group_alias at match time on both sides; a per-key or per-team alias is not consulted

Low

• A request outside the model scope records no funnel event, matching how a direction mismatch is treated
• models is validated as "could any caller on this proxy ask for this name" (unscoped resolution or a team-public name), never per team; an auto-router name is allowed, since a forward job on router A scoped to router B's group is a real question

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Adds a DB column and changes shadow-eval admission on the success-hook hot path; mis-scoping or alias behavior could skew samples or spend, though empty models preserves legacy behavior.

Overview
Shadow eval jobs can now be narrowed with an optional models list that ANDs with existing key, team, and user targets. An empty list keeps today’s behavior (all model groups for those targets).

Persistence and API: LiteLLM_ShadowEvalJob gains a models text array (migration + Prisma). Start/list/detail bodies accept and return models; values are deduped and must be non-empty names. models on reverse jobs is rejected at validation time. At start, _validate_model_scope returns 400 if a name is not something any caller on the proxy could request (including team-public and auto-router names where applicable).

Sampling: ShadowEvalLogger only admits traffic when the request’s model_group matches the job scope after resolve_model_group_alias on both sides at match time. Out-of-scope requests are ignored with no funnel “not_sampled” event, same as a direction mismatch.

Dashboard: Forward jobs get an “Only on models” multi-select; job headlines show the scope when set. OpenAPI/schema.d.ts updated.

^{Reviewed by Cursor Bugbot for commit 8000cfb. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_shadow_eval_model_scope (8000cfb) with litellm_internal_staging (59d42d3)}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[greptile-apps]

Greptile Summary

This PR adds an optional model-group scope to forward shadow-evaluation jobs and composes it with existing key, team, and user targeting.

• Persists and exposes model scopes through the database, API models, and generated dashboard types.
• Filters admitted traffic by the requested model group while resolving aliases at match time.
• Validates model names at job creation and rejects model scopes for reverse jobs.
• Adds a dashboard model picker, scoped-job headline text, API tests, logger tests, and UI tests.
• The three previous findings are resolved: aliases are now resolved per request, team-public names pass proxy-wide validation, and the form enforces the API’s 100-model limit.

Confidence Score: 5/5

The PR appears safe to merge; the previously reported defects are resolved and no new actionable issue remains.

Alias canonicalization now occurs at request matching time, proxy-wide validation accepts backed team-public names, reverse jobs reject model scopes, and the dashboard prevents submission above the API’s 100-model limit. All previous threads were also manually resolved.

Important Files Changed

Filename	Overview
litellm/integrations/shadow_eval_logger.py	Adds model-scope admission and resolves aliases on both the request and job scope at match time.
litellm/proxy/management_endpoints/auto_router_endpoints.py	Validates proxy-requestable model scopes and persists and returns them for every shadow-eval leg.
litellm/types/management_endpoints/auto_router_endpoints.py	Adds bounded, deduplicated model-scope request and response fields and disallows scopes on reverse jobs.
litellm-proxy-extras/litellm_proxy_extras/migrations/20260904000000_shadow_eval_model_scope/migration.sql	Adds a non-null model-scope array with an empty legacy-compatible default.
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalStartForm.tsx	Adds the forward-only model picker, enforces the API limit, and serializes the selected scope.
ui/litellm-dashboard/src/app/(dashboard)/cost-optimization/_components/ShadowEvalSection.tsx	Displays configured model scopes in shadow-evaluation job headlines.
tests/test_litellm/integrations/test_shadow_eval_logger.py	Covers scoped and unscoped admission, aliases in either spelling, and immediate alias repointing.
tests/test_litellm/proxy/management_endpoints/test_auto_router_endpoints.py	Covers persistence, response behavior, validation, deduplication, team-public names, and reverse-job rejection.

_{Reviews (2): Last reviewed commit: "feat(shadow_eval): scope a job to model ..." | Re-trigger Greptile}

[tin-berri]

CI is 192/193 green on 5d6d7b1. The one red, proxy-infra Python 3.10 test_qualifiers_and_optionality_are_unwrapped, fails identically on staging's own run at this branch's merge base d23bec8 (run 33925150414), so it predates the branch and is not from this diff

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Confidence Score: 3/5

The core feature is well wired: the model scope is persisted on every leg, returned by the API, applied as an AND filter in the logger, alias matching is covered in both directions, unscoped jobs remain compatible, and the dashboard/tests cover the main user flow. The reported CI result also indicates the only failing check is pre-existing.

I’m not raising this to 4/5 because the two P1 correctness issues remain:

• In litellm/integrations/shadow_eval_logger.py, model scopes are canonicalized when active jobs are loaded and then retained in the jobs cache. If model_group_alias is repointed while that cache is warm, requests are canonicalized with the new mapping but cached jobs still use the old mapping, so traffic can be admitted to the wrong scope until cache expiry.
• In litellm/proxy/management_endpoints/auto_router_endpoints.py, _validate_model_scope() validates only team_ids. User-target jobs can span teams, so a model group valid through a user’s team can be rejected at start even though matching traffic can carry that model name.

There is also a smaller UX issue: the dashboard picker does not enforce the API’s 100-model limit, allowing a submission that the backend will reject. Fixing the two P1 issues would make this a 4/5; the remaining implementation and test coverage are otherwise strong.

[cursor]

Stale Bugbot comment from a previous run.

[tin-berri]

Codecov's final pull API now reports 100.0% patch coverage (27 hits, 0 misses) for 8000cfb. The prior codecov/patch failure was a stale status; the only remaining CI red is the already-documented inherited Python 3.10 proxy-infra test

[tin-berri]

@greptileai can you reply here with a score and why that score?

[tin-berri]

bugbot run

[greptile-apps]

Confidence Score: 4/5

The two correctness issues from the previous review are addressed:

• Alias canonicalization now happens against the current router alias map during admission, while the cached job retains the raw scope. The added repointing test verifies that a warm active-job cache follows alias changes correctly.
• _validate_model_scope() now permits team-public model names independently of the selected target teams, which is required for user-target jobs spanning multiple teams. This is covered by a regression test.

The implementation also persists and returns models consistently, preserves unscoped-job behavior, rejects invalid/reverse/model-only scopes, and includes backend, logger, and dashboard coverage. The dashboard now prevents submissions over the API’s 100-model limit as well.

I’m keeping this at 4/5 rather than 5/5 because one CI check remains red, although the provided evidence indicates that failure is inherited from the merge base rather than introduced here. From the feature-correctness perspective, I have no remaining blocking issue.

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 8000cfb. Configure here.}

[tin-berri]

@greptileai review