Repository navigation
fix(proxy): route container create and list through model_list deployments - #39220
mateo-berri merged 3 commits into
Conversation
…ments Container create and list requests had no container ID to decode, so the router called the provider handler directly and the OpenAI transformation fell back to the global OPENAI_API_KEY. Proxies configured only with model_list credentials sent Authorization: Bearer None. Route through _ageneric_api_call_with_fallbacks when the caller passes a model, expose the list endpoint's model query param to the router, and encode the managed container ID on the async create path so follow-up calls route to the same deployment. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
|
|
No action taken: PR #39220 (BerriAI/litellm, author devin-ai-integration[bot]) has no labels and lacks the |
Greptile SummaryThe PR routes container creation and listing through configured model deployments and preserves deployment metadata in IDs returned by asynchronous creation
Confidence Score: 5/5The PR appears safe to merge No blocking failure remains
|
| Filename | Overview |
|---|---|
| litellm/containers/main.py | Adds post-await managed-ID encoding for asynchronous container creation |
| litellm/proxy/container_endpoints/endpoints.py | Passes the list endpoint’s model query parameter into request processing |
| litellm/router.py | Routes model-bearing container create and list operations through configured deployments |
| tests/router_unit_tests/test_router_endpoints.py | Adds coverage for configured, absent, and unknown deployment routing |
| tests/test_litellm/containers/test_container_api.py | Adds functional coverage for managed-ID encoding after asynchronous creation |
Reviews (4): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…eployment Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
…itellm_containers_route_model_list_creds
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 4a09c35. Configure here.
TLDR
Problem this solves:
model_listcredentialsAuthorization: Bearer Noneto OpenAI without a global keyHow it solves it:
modelmodelthat names no deployment falls through to the direct call, as beforemodelfrom the query stringcntr_...ID like sync doesUser Flow
Before: a team that only stores OpenAI credentials in
model_listcan chat through the gateway but cannot touch the Containers API"model": "gpt-5.4"and get a 200 with a completion{"name": "demo", "model": "gpt-5.4"}Incorrect API key provided: NoneAfter: the same team creates, lists, retrieves and deletes containers through the gateway
"model": "gpt-5.4"and get a 200 with a completion{"name": "demo", "model": "gpt-5.4"}cntr_and is longer than OpenAI's ownRelevant issues
Reported by a customer (Pylon #8049)
Linear ticket
Resolves LIT-6667
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*,make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more@greptileaito re-request a review after pushing changes)Delays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
Screenshots / Proof of Fix
Real OpenAI API,
OPENAI_API_KEYunset in the proxy environment, the only OpenAI credential inmodel_list. Two proxies booted with--num_workers 2, each on its own Postgres: before at the merge base a677242, after at this PR's tip 4a09c35. Same config, same calls, same orderBefore (a677242, port 40540)
After (4a09c35, port 33458)
Observations from the run:
Type
🐛 Bug Fix
Caveats (if any)
Low
modelstill needs a globalOPENAI_API_KEYmodel_listmodel now returns a managedcntr_...ID instead of the raw upstream one, matching the Responses API. Every container endpoint accepts both forms/v1/responsesand/v1/filesdo;num_retries: 0turns that offlimitand other pagination params; pre-existing, untouched hereFinal Attestation
Link to Devin session: https://app.devin.ai/sessions/25f80f570f9b43fe82383d92e8dacdbf
Open in Devin Desktop: https://app.devin.ai/desktop/session/25f80f570f9b43fe82383d92e8dacdbf?variant=devin
Note
Medium Risk
Changes proxy credential routing and container ID shape for model-routed creates; behavior is covered by new tests and mirrors existing files/responses routing, but touches auth-sensitive gateway paths.
Overview
Fixes proxy Containers API auth and routing when OpenAI credentials live only in
model_list, not in a globalOPENAI_API_KEY.Router:
_init_containers_api_endpointsnow sends create/list (no managedcntr_...ID) through_ageneric_api_call_with_fallbackswhen the caller passes a non-emptymodel, injecting that deployment’sapi_key/api_base. Unknown model names usepassthrough_on_no_deployment=Trueso the handler is still invoked directly without a 400, same as before.Proxy: GET
/v1/containersforwardsmodelfrom the query string into the request payload so list can pick a deployment.SDK:
create_containerapplies managed-ID encoding after async upstream responses via_encode_created_container_id, so routed creates return embeddablecntr_...IDs for retrieve/delete/files instead of raw upstream IDs.Reviewed by Cursor Bugbot for commit 4a09c35. Bugbot is set up for automated code reviews on this repo. Configure here.