feat(decisions): add Laya and Bespoke providers to the Add Model form

[devin-ai-integration]

TLDR

Problem this solves:

• Laya and Bespoke Nimble can't be added from the Add Model form
• laya/* and bespoke/* deployments aren't served by the Decisions routes at all

How it solves it:

• Adds Laya and Bespoke Nimble to the provider fields and the dashboard provider maps
• Registers laya and bespoke as Decisions providers calling <api_base>/v1/systemone
• Adds laya and bespoke rows to the decisions wire integration test
• The Decisions - /v1/decisions mode is already on main, so this PR does not carry it
• Laya and Bespoke send LAYA_API_KEY or BESPOKE_API_KEY only to a deployment or Auto Router classifier on LAYA_API_BASE or BESPOKE_API_BASE, and a key typed in the form always wins
• An API base or env base ending in /v1 calls the same /v1/systemone from a deployment, an Auto Router classifier, and the /laya and /bespoke pass-through routes, where main's classifier and pass-through called /v1/v1/systemone
• The Bespoke API Base field's tooltip says to pick bespoke/nimble on Ollama
• An unknown checkpoint or an API base with credentials, a query, or a non-HTTP scheme answers 400

Decisions this PR makes. The env key goes to a deployment or an Auto Router classifier only when its API base is the env base, ignoring a trailing / or /v1 since both call the same URL. Main sent it only when no API base was set, so a classifier whose explicit api_base repeats the env base now sends the key to that server where main sent none. Sending it to any base would hand the operator's key to whatever server a deployment names, the way OPENAI_API_KEY goes to an explicit api_base, and never sending it to an explicit base would leave the env key unusable from the form, which always sets one. The env var docs change with this in BerriAI/litellm-docs#2356. laya/* and bespoke/* are priced at 0 in the cost map, the same as Strands Decider, because they are self-hosted and nobody bills per token. An admin who wants spend sets Custom Pricing on the deployment, which the QA below shows, while a non-zero default would bill a cost nobody charges. The checkpoint allowlist matches the cost-map rows, so an alias like laya/auto or an Ollama tag like nimble:latest answers 400 instead of reaching a server under a name with no price

User Flow

Before: an admin who self-hosts Laya or Bespoke Nimble can't add it from the Add Model form, and the model can't be called even when added by hand

1. They open http://localhost:4000/ui/models-and-endpoints/?tab=add-model, click the Add Model tab, and type Laya or Bespoke into the Provider dropdown: "No providers found"
2. They fall back to POST http://localhost:4000/model/new with "model": "laya/english" and their API base: 500, "model id(s) ... are not live in this pod's router", the same for bespoke/nimble
3. POST http://localhost:4000/v1/systemone and POST http://localhost:4000/v1/decisions with "model": "laya/english" answer 400 "Invalid model name passed in", the same for bespoke/nimble

After: the admin adds both from the form, calls them straight away, and sees the spend they priced

1. They open the same page and click the Add Model tab. The Provider dropdown now lists Laya and Bespoke Nimble
2. They pick Laya, choose laya/english, set Mode to Decisions - /v1/decisions, enter the API base and API key, turn on Custom Pricing under Advanced Settings with an input and an output cost, and click Add Model: "Model laya-form-priced created successfully"
3. POST http://localhost:4000/v1/systemone (System One body) and POST http://localhost:4000/v1/decisions (OpenAI body) with that model answer 200 with a non-zero x-litellm-response-cost, and the calls show their cost on http://localhost:4000/ui/logs
4. They pick Bespoke Nimble, read the API Base tooltip that says to pick bespoke/nimble on Ollama, and the added model answers on both routes with its token usage
5. A deployment on the operator's LAYA_API_BASE, written with or without a trailing / or /v1, works with the API Key field left empty, because the proxy sends LAYA_API_KEY there and nowhere else
6. TypeSafe, Perplexity, OpenRouter, and Strands Decider still add from the same form and answer on both routes with a non-zero cost
7. A deployment with an unknown checkpoint or an API base carrying a query string answers 400 naming the problem

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• Every automated review on my PR's latest commit has finished, and I have addressed or replied to each of its findings before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Last updated: d4d74f0. QA ran on d4d74f0, the last commit with a product diff

Shared setup, identical on both sides apart from the commit the proxy and the dashboard were built from

• One proxy with --num_workers 2, STORE_MODEL_IN_DB=True, an empty model_list, LITELLM_LOCAL_MODEL_COST_MAP=True, its own empty Postgres database, and the dashboard dev server from the same tree rewriting to it, so the Add Model form is the real one. The after side also sets LAYA_API_BASE and LAYA_API_KEY to the keyed Laya server
• Real local upstreams. Laya serve 0.4.1 with a key on http://127.0.0.1:48029 (401 without it), Laya serve with no key on http://127.0.0.1:47179, Ollama on http://127.0.0.1:35867 serving Bespoke Nimble, and Strands Decider on http://127.0.0.1:48559. TypeSafe, Perplexity, and OpenRouter are the hosted APIs
• System One body {"model":"<model>","state":"ticket: export hangs at 99%","questions":{"is_defect":{"type":"noul","instructions":"Is this a defect?"}}} and OpenAI body {"model":"<model>","input":"ticket: export hangs at 99%","questions":[{"type":"predicate","name":"is_defect","instructions":"Is this a defect?"}]}
• Every form case signs in at http://localhost:/ui/login, opens http://localhost:/ui/models-and-endpoints/?tab=add-model, and clicks the Add Model tab. Responses are trimmed to the status line, the cost headers, and the body

Before (e226016, the merge base)

1. GET /public/providers/fields lists 132 providers with typesafe and strands_decider among them, and no laya or bespoke
2. Typing Laya or Bespoke in the Provider dropdown shows "No providers found" (screenshots in User Flow)
3. Adding the deployments by hand saves the row and then fails

$ curl -s -w '\nHTTP %{http_code}\n' -X POST http://127.0.0.1:45040/model/new -H 'Authorization: Bearer $MASTER' -H 'Content-Type: application/json' \
  -d '{"model_name":"laya/english","litellm_params":{"model":"laya/english","api_base":"http://127.0.0.1:48029","api_key":"$KEY"},"model_info":{"mode":"decisions"}}'
{"error":{"message":"Model create was saved to the database, but the model id(s) ['7ed37801-...'] are not live in this pod's router after the reload and are not being served by this pod. ...","type":"internal_server_error",...}}
HTTP 500

bespoke/nimble on http://127.0.0.1:35867 answers the same 500, and GET /v1/models returns {"data":[],"object":"list"}

4. Both Decisions routes refuse both models

$ curl -s -D - -X POST http://127.0.0.1:45040/v1/systemone -H 'Authorization: Bearer $MASTER' -H 'Content-Type: application/json' -d '<System One body, laya/english>'
HTTP/1.1 400 Bad Request
x-litellm-response-cost: 0
{"error":{"message":"/systemone: Invalid model name passed in model=laya/english. Call `/v1/models` to view available models for your key.","type":"invalid_request_error","param":null,"code":"400"}}

$ curl -s -D - -X POST http://127.0.0.1:45040/v1/decisions ... -d '<OpenAI body, laya/english>'
HTTP/1.1 400 Bad Request
x-litellm-response-cost: 0
{"error":{"message":"/decisions: Invalid model name passed in model=laya/english. ...","type":"invalid_request_error","code":"400"}}

bespoke/nimble answers the same two 400s

After (d4d74f0)

A. Laya with the key typed in the form and Custom Pricing of 1 input and 2 output per 1M tokens. The toast reads "Model laya-form-priced created successfully" and the form sends

POST /model/new {"model_name":"laya-form-priced","litellm_params":{"model":"laya/english","custom_llm_provider":"laya","api_base":"http://127.0.0.1:48029","api_key":"<redacted>","input_cost_per_token":1e-06,"output_cost_per_token":2e-06,"cache_read_input_token_cost":1e-06},"model_info":{"mode":"decisions"}}   -> 200

$ curl -s -D - -X POST http://127.0.0.1:39855/v1/systemone -H 'Authorization: Bearer $MASTER' -H 'Content-Type: application/json' -d '<System One body, laya-form-priced>'
HTTP/1.1 200 OK
x-litellm-model-api-base: http://127.0.0.1:48029
x-litellm-response-cost: 3.7999999999999995e-05
{"model":"laya-rl-agent","answers":{"is_defect":{"type":"noul","noul":0.8582,...}},"usage":{"input_tokens":38,"output_tokens":0,...},"routing":{"model":"english",...}}

$ curl -s -D - -X POST http://127.0.0.1:39855/v1/decisions ... -d '<OpenAI body, laya-form-priced>'
HTTP/1.1 200 OK
x-litellm-response-cost: 3.7999999999999995e-05
{"model":"laya-rl-agent","answers":[{"type":"predicate","name":"is_defect","probability":0.8582}],"usage":{"input_tokens":38,...,"total_tokens":38}}

The cost is 38 input tokens at $1 per 1M. The server reports 0 output tokens, so the output price adds nothing

B. Laya on the env server with the API Key field empty and a trailing slash on the API base. The form sends {"model":"laya/english","custom_llm_provider":"laya","api_base":"http://127.0.0.1:48029/"} with no api_key, and both routes answer 200 from the server that answers 401 without its key

$ curl -s -o /dev/null -w '%{http_code}\n' -X POST http://127.0.0.1:48029/v1/systemone -H 'Content-Type: application/json' -d '{"model":"english","state":"x","questions":{"q":{"type":"noul","instructions":"y"}}}'
401

$ curl -s -D - -X POST http://127.0.0.1:39855/v1/systemone ... -d '<System One body, laya-env-base>'
HTTP/1.1 200 OK
x-litellm-model-api-base: http://127.0.0.1:48029/
{"model":"laya-rl-agent","answers":{"is_defect":{"type":"noul","noul":0.8582,...}},"usage":{"input_tokens":38,...}}

/v1/decisions answers the same 200

B2. The same env server written as http://127.0.0.1:48029/v1, API Key field empty. The form sends {"model":"laya/english","custom_llm_provider":"laya","api_base":"http://127.0.0.1:48029/v1"} with no api_key, and both routes answer 200 (screenshot in User Flow)

$ curl -s -D - -X POST http://127.0.0.1:39855/v1/systemone ... -d '<System One body, laya-env-v1>'
HTTP/1.1 200 OK
x-litellm-model-api-base: http://127.0.0.1:48029/v1
{"model":"laya-rl-agent","answers":{"is_defect":{"type":"noul","noul":0.8582,...}},"usage":{"input_tokens":38,...}}

/v1/decisions answers the same 200, and the keyed server's log shows one POST /v1/systemone per call, never /v1/v1/systemone

C. Laya on the other server with no key (laya-open, API base http://127.0.0.1:47179) answers 200 on both routes

D. Bespoke Nimble. GET /public/providers/fields lists bespoke as "Bespoke Nimble" with placeholder bespoke/nimble, and the form's model list offers bespoke/nimble-latest, bespoke/nimble, and bespoke/bespokelabs/Bespoke-Nimble-9B. The form sends {"model":"bespoke/nimble","custom_llm_provider":"bespoke","api_base":"http://127.0.0.1:35867"} and both routes answer

$ curl -s -D - -X POST http://127.0.0.1:39855/v1/systemone ... -d '<System One body, bespoke/nimble>'
HTTP/1.1 200 OK
x-litellm-model-api-base: http://127.0.0.1:35867
{"model":"nimble","answers":{"is_defect":{"type":"noul","noul":0.7822738692608625}},"usage":{"input_tokens":158,"output_tokens":0}}

$ curl -s -D - -X POST http://127.0.0.1:39855/v1/decisions ... -d '<OpenAI body, bespoke/nimble>'
HTTP/1.1 200 OK
{"model":"nimble","answers":[{"type":"predicate","name":"is_defect","probability":0.7822738692608625}],"usage":{"input_tokens":158,...,"total_tokens":158}}

T. With Bespoke Nimble picked, resting the pointer on the API Base help icon shows "On Ollama, pick bespoke/nimble as the model" (screenshot in User Flow), the same text the form reads from the API

$ curl -s http://127.0.0.1:39855/public/providers/fields | jq -c '.[] | select(.litellm_provider == "bespoke") | [.credential_fields[] | select(.key == "api_base") | {key, tooltip}]'
[{"key":"api_base","tooltip":"On Ollama, pick bespoke/nimble as the model"}]

E. The decision providers already in the form, each added through it with Mode Decisions - /v1/decisions. Strands Decider has Custom Pricing of 0.5 input and 1.5 output per 1M, and the other three use their bundled prices

Provider	Form add	/v1/systemone	/v1/decisions	Cost per call
TypeSafe	200	200	200	1.176e-05
Perplexity	200	200	200	1.86e-06
OpenRouter	200	200	200	1.176e-05
Strands Decider	200	200	200	4.3e-05

Strands Decider's two calls, on the same boot as the rest

$ curl -s -D - -X POST http://127.0.0.1:39855/v1/decisions ... -d '<OpenAI body, strands_decider/strands-decider-2B-hobson-v19>'
HTTP/1.1 200 OK
x-litellm-model-api-base: http://127.0.0.1:48559
x-litellm-response-cost: 4.3e-05
x-litellm-response-cost-input: 4.15e-05
x-litellm-response-cost-output: 1.5e-06
{"model":"strands-decider-2B-hobson-v19","answers":[{"type":"predicate","name":"is_defect","probability":0.736}],"usage":{"input_tokens":83,...,"output_tokens":1,...,"total_tokens":84}}

F. Bad deployment values save through /model/new and answer 400 at call time

$ curl -s -D - -X POST http://127.0.0.1:39855/v1/systemone ... -d '<System One body, a laya/any-checkpoint deployment>'
HTTP/1.1 400 Bad Request
{"error":{"message":"litellm.BadRequestError: laya model must be one of english, multilingual, typed-decisions ...","type":"invalid_request_error","code":"400"}}

$ curl -s -D - -X POST http://127.0.0.1:39855/v1/systemone ... -d '<System One body, laya/english with api_base http://127.0.0.1:47179?token=x>'
HTTP/1.1 400 Bad Request
{"error":{"message":"litellm.BadRequestError: laya api_base must not contain credentials, a query, or a fragment ...","type":"invalid_request_error","code":"400"}}

http://localhost:/ui/logs shows the laya-form-priced calls at $0.000038, TypeSafe and OpenRouter at $0.000012, Perplexity at $0.000002, and the unpriced Laya and Bespoke calls at "-" (screenshot in User Flow)

Things the run noticed

• Zero-cost 200s drop the bare cost header, main too
• Laya and Bespoke always report 0 output tokens
• /model/new saves bad checkpoints, rejected at call time
• ?tab=add-model opens Deployed Models, not Add Model
• Bespoke model field is a select, so placeholder never shows
• Logs Model column shows laya/english, not the public name
• Unsynced worker 429s a just-added model, not this PR
• The api base header keeps /v1, the call drops it

/live-pr-risk (d4d74f0 against the merge base e226016)

Each side ran 2 workers on its own Postgres, with real Laya (keyed and keyless), Ollama Bespoke Nimble and Strands Decider upstreams behind forwarding recorders, and four boots per side for LAYA_API_BASE set to a plain base, a /v1 base, http://v1 and http://v1/v1. Every cell made exactly one upstream attempt on both sides

Breaking. None. The first run found that a base whose host is named v1 (http://v1) lost its host in the /v1 strip, and d4d74f0 fixed it by stripping /v1 from the URL path only. Classifiers and the /laya pass-through now reach a v1 host on every cell, and 2,073 generated bases (IPv6, ports, percent-encoding, upper case) come back as the checked base minus a trailing /v1 path

Backward incompatible. An explicit classifier api_base that names the LAYA_API_BASE or BESPOKE_API_BASE server now gets that server's env key. Base sent no key there and fell back to the heuristic scorer after a 401, and head classifies. Every Auto Router call answered 200 on both sides, and only the router cause header and one extra $0 classifier spend row changed. This follows the OPENAI_API_KEY convention in litellm/main.py, where an env key goes with an explicit api_base, narrowed here to the env server, and shipped in e3452c5. The other server never got the env key in any cell (deployments, health checks, classifiers). A classifier or pass-through on a /v1 base now calls /v1/systemone once, where base called /v1/v1/systemone and got 404

Regression risk. These paths are untested and traced only, laya/english matching a fallback entry keyed english, _infer_valid_provider_from_env_vars counting LAYA_API_KEY, and the rate limiter labels. A laya health check writes a $0 acompletion spend row, the same kind Strands' health check writes on base

Dependency graph. These were verified live on both sides, the classifier client (Auto Router chat and /auto_router/test_routing), both pass-through routes, /v1/systemone, /v1/decisions, /model/new, /health, /public/providers/fields, /model/settings, the guardrail decision model list, and main's AI Hub garden on the merged tree. The merge into current main is clean and matched head on 27 of 28 cells. The one difference is main's #45734, where a laya call with images now answers 400 unless drop_params is set

Not verified. BESPOKE_API_BASE=http://v1 was not booted, since bespoke shares laya's connection code. The bespoke Auto Router was left out of the second run because Ollama took about 4 minutes per call under load, and it matched the first run's key difference

• d4d74f0 passes /audit (skipped: new decisions provider plus Add Model form entries, one endpoint family)

Type

🆕 New Feature

Caveats (if any)

Low

• Laya and Bespoke have no logo asset, so they show a letter avatar
• laya/* and bespoke/* cost $0 unless the deployment sets Custom Pricing, so their spend rows show real tokens and $0. The $0 rows are main's, like ollama/*
• An Auto Router classifier whose api_base names the LAYA_API_BASE or BESPOKE_API_BASE server now sends that server's env key. Main sent it only when api_base was left unset, and docs(decisions): add Laya and Bespoke Nimble as decisions providers litellm-docs#2356 updates the docs
• Zero-cost 200s carry the x-litellm-response-cost-* breakdown headers but no bare x-litellm-response-cost, on main too
• Test Connect on a decisions model runs a chat health check, which is main's form code and unchanged here
• /v1/decisions ignores stream and custom_body for every decisions provider, on main too, and this PR does not touch that route
• An alias like laya/auto or an Ollama tag like nimble:latest answers 400, since the allowlist matches the cost-map rows
• bespoke/nimble-latest and bespoke/bespokelabs/Bespoke-Nimble-9B did not run live, since the QA box has no server for those names. They take the same code path as bespoke/nimble, which ran
• A Decisions deployment whose API base host is named v1 (http://v1) gets 404 for every Decisions provider, Strands included, because main's shared URL builder strips /v1 from the whole string. Laya and Bespoke inherit it, while their classifiers and pass-through routes reach such a host
• /model/new saves a bad checkpoint or API base, and the 400 shows at the first call, the same as for any provider on main
• The laya and bespoke rows of the decisions wire integration test run only in main's CircleCI, not in this PR's checks
• The Mode entry reads Decisions - /v1/decisions, which main added, so this PR adds no separate evaluation mode

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

• d4d74f0 passes /live-pr-risk

---

Note

Medium Risk
Changes how env API keys are paired with API bases for OSS Decisions classifiers and adds new routing paths, though behavior is heavily tested and keys are not sent to arbitrary deployment URLs.

Overview
Adds Laya and Bespoke Nimble as first-class Decisions providers so laya/* and bespoke/* models can be configured in the proxy Add Model form and routed to <api_base>/v1/systemone with allowlisted checkpoints and optional API keys.

Decisions connection setup now goes through resolve_connection on BaseDecisionsConfig (replacing separate base/key resolution in decisions/call.py). Laya and Bespoke use shared oss_connection logic that normalizes API bases (including stripping a trailing /v1 so requests do not hit /v1/v1/systemone) and only attaches LAYA_API_KEY / BESPOKE_API_KEY from the environment when the target server matches the configured env base—explicit deployment bases elsewhere do not receive the operator env key unless a key is passed in the request.

Provider registration, lazy imports, proxy provider_create_fields.json, and dashboard provider maps/placeholders are updated; integration and unit tests cover proxy dispatch, env-key scoping, and validation of unknown models and malformed API bases.

^{Reviewed by Cursor Bugbot for commit d4d74f0. Bugbot is set up for automated code reviews on this repo. Configure here.}

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ mateo-berri
❌ litellm-stampbot
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge, with both previous findings addressed and no new blocking issues

Summary

Adds Laya and Bespoke Nimble to the Add Model form and connects both providers to the Decisions routes

• Admins can pick Laya and Bespoke Nimble in the Add Model form.
• LiteLLM routes Laya and Bespoke Nimble calls through Decisions.
• The cost suite checks TypeSafe calls to /v1/systemone.

_{Reviews (3) · Last reviewed commit: "test(proxy): type the OSS decisions prov..." · Reviewed by Greptile}

[codspeed]

Merging this PR will not alter performance

⚠️ 1 benchmark spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

• test_completion_streaming

✅ 31 untouched benchmarks

---

_{Comparing litellm_add_model_decision_providers (d4d74f0) with main (aa0279e)}

[codecov]

Codecov Report

❌ Patch coverage is 95.74468% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/llms/oss_decision.py	87.50%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

[mateo-berri]

#45335 is merged and already adds the TypeSafe and Strands Decider entries, so this PR needs a rebase that drops those two

[devin-ai-integration]

@mateo-berri I merged main and dropped our TypeSafe and Strands Decider copies, so main's versions win. What's left in this PR is Laya, Bespoke, the Evaluation mode and the TypeSafe cost case

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[mateo-berri]

@litereview code-review

[litereview]

✅ code-review

5/5 · Ready for human review

Scope: full · Full PR review.

No correctness findings.

Checks and execution details

• Passed · Real testing evidence (required): In scope. The PR adds litellm/llms/laya/decisions/transformation.py and litellm/llms/bespoke/decisions/transformation.py, and it changes litellm/llms/oss_decision.py and litellm/llms/base_llm/decisions/transformation.py. Those changes alter request routing (<api_base>/v1/systemone), checkpoint validation, and when the auth header is sent.

Manual E2E evidence. The PR description includes inspectable manual E2E proof run on db83640aa8:

• Environment: a real proxy (2 workers, Postgres, dashboard dev server) talking to real upstreams. Upstreams were a keyed Laya serve 0.4.1 server that returns 401 without its key, an unkeyed Laya server, and Ollama serving Bespoke Nimble.
• Steps: each model was added through the Add Model form, then called through /v1/systemone and /v1/decisions.
• Observed results:
  • Both routes returned 200 with costs, using the bodies shown in the description.
  • With the API Key field empty, the env key reached the server named by LAYA_API_BASE, including when the base had a trailing /.
  • An unknown checkpoint and an API base with a query string each returned 400.

Gap between QA and head. Head d4d74f0 adds one commit after the QA run. GitHub compare db83640..d4d74f0 shows it changes only _server_root: /v1 is now stripped from the URL path, so a host literally named v1 keeps its name. The new test_oss_decision.py rows cover that change.

Tests I ran locally (all passed): uv run --frozen --all-extras pytest on tests/unit/llms/test_oss_decision.py, tests/unit/decisions/test_main.py, tests/unit/proxy/decisions_endpoints/test_endpoints.py, tests/unit/router_strategy/complexity_router/test_jev_classifier.py and tests/unit/proxy/public_endpoints/test_public_endpoints.py: 449 passed.

CI on head: GitHub check-runs for d4d74f0 all report success, including unit, llms-providers, router, proxy-endpoints and ui-unit.

Not relied on: the new laya/bespoke rows in tests/integration/providers/test_decisions_wire.py use synthetic upstream scenarios. The PR says they run only in main's CircleCI, so they are not part of this evidence.

• Passed · Changed-behavior trace (optional): I traced how each caller reaches oss_connection:

• Decisions calls: decisions/call.py::prepare_call calls provider_config.resolve_connection, and ValueError maps to 400. The other providers use the default resolve_connection in BaseDecisionsConfig, which is equivalent to the old resolve_api_base and resolve_api_key pair.

• Pass-through routes: /laya and /bespoke go through llm_passthrough_endpoints.py::_oss_decision_proxy_route, which calls oss_connection(provider) with the env base and therefore gets the env key.

• Auto Router classifier: complexity_router._build_jev_client now sends the env key when an explicit api_base matches the env base. The PR documents that change on purpose.

No other callers of resolve_api_base or resolve_api_key are affected. Cost-map rows for all six allowlisted checkpoints already exist on main.

• Passed · Breaking-change check (optional): I found no breaking change to a supported API, default, or persisted data.

• Pass-through URL: the move from /v1/v1/systemone to /v1/systemone fixes a bug.

• Auto Router env key: sending the env key to a classifier whose explicit api_base repeats LAYA_API_BASE widens behavior. The key still goes only to the operator's configured server, and docs(decisions): add Laya and Bespoke Nimble as decisions providers litellm-docs#2356 documents the change. The title does not need a ! marker.

---

View review · Reviewed d4d74f0bd91c

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@litereview code-review

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@litereview code-review

[mateo-berri]

@litereview code-review

[mateo-berri]

@litereview code-review

[mateo-berri]

@litereview code-review

[mateo-berri]

bugbot run

[cursor]

Stale Bugbot comment from a previous run.

[mateo-berri]

@litereview code-review

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit d4d74f0. Configure here.}

[mateo-berri]

LGTM