Skip to content

fix(proxy): route container create and list through model_list deployments - #39220

Merged
mateo-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_containers_route_model_list_creds
Sep 2, 2026
Merged

mateo-berri merged 3 commits into
litellm_internal_stagingfrom
litellm_containers_route_model_list_creds

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • Container create and list ignored model_list credentials
  • Proxy sent Authorization: Bearer None to OpenAI without a global key
  • Async create returned raw upstream IDs, so follow-ups lost routing info

How it solves it:

User Flow

Before: a team that only stores OpenAI credentials in model_list can chat through the gateway but cannot touch the Containers API

  1. They send POST https://litellm-domain/v1/chat/completions with "model": "gpt-5.4" and get a 200 with a completion
  2. They send POST https://litellm-domain/v1/containers with {"name": "demo", "model": "gpt-5.4"}
  3. They get a 401 with Incorrect API key provided: None
  4. They send GET https://litellm-domain/v1/containers?model=gpt-5.4 and get a 500 wrapping the same auth error

After: the same team creates, lists, retrieves and deletes containers through the gateway

  1. They send POST https://litellm-domain/v1/chat/completions with "model": "gpt-5.4" and get a 200 with a completion
  2. They send POST https://litellm-domain/v1/containers with {"name": "demo", "model": "gpt-5.4"}
  3. They get a 200 with a gateway-managed ID that starts with cntr_ and is longer than OpenAI's own
  4. They send GET https://litellm-domain/v1/containers?model=gpt-5.4 and get a 200 with the account's containers
  5. They send GET and DELETE https://litellm-domain/v1/containers/ and both return 200

Relevant issues

Reported by a customer (Pylon #8049)

Linear ticket

Resolves LIT-6667

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Real OpenAI API, OPENAI_API_KEY unset in the proxy environment, the only OpenAI credential in model_list. Two proxies booted with --num_workers 2, each on its own Postgres: before at the merge base a677242, after at this PR's tip 4a09c35. Same config, same calls, same order

model_list:
  - model_name: gpt-5.6
    litellm_params:
      model: openai/gpt-5.6
      api_key: os.environ/MY_OPENAI_KEY

general_settings:
  master_key: sk-lit6667
env -u OPENAI_API_KEY python litellm/proxy/proxy_cli.py --config lit6667_config.yaml --port <port> --num_workers 2 --use_v2_migration_resolver --detailed_debug

Before (a677242, port 40540)

KEY_A and KEY_B are two non-admin virtual keys from POST /key/generate (models: [gpt-5.6]), each with its own user_id

$ curl -s -X POST http://127.0.0.1:40540/v1/chat/completions -H 'Authorization: Bearer $KEY_A' -H 'Content-Type: application/json' -d '{"model":"gpt-5.6","messages":[{"role":"user","content":"say hi"}],"max_tokens":5}' -w '\nHTTP %{http_code}\n' | jq -c '{model, content: .choices[0].message.content} ...
{"model":"gpt-5.6","content":""}
HTTP 200

$ curl -s -X POST http://127.0.0.1:40540/v1/containers -H 'Authorization: Bearer $KEY_A' -H 'Content-Type: application/json' -d '{"name":"lit6667-before","model":"gpt-5.6"}' -w '\nHTTP %{http_code}\n'
{"error":{"message":"litellm.AuthenticationError: AuthenticationError: OpenAIException - {\n  \"error\": {\n    \"message\": \"Incorrect API key provided: None. You can find your API key at https://platform.openai.com/account/api-keys.\",\n    \"type\": \"invalid_request_error\",\n    \"param\": nul ...
HTTP 401

$ sleep 3  # OpenAI lists a new container about 1-2s after create (same lag when calling api.openai.com directly)

$ curl -s 'http://127.0.0.1:40540/v1/containers?model=gpt-5.6&limit=5' -H 'Authorization: Bearer $KEY_A' -w '\nHTTP %{http_code}\n' | jq -c '{object, ids: [.data[].id], has_more}'
{"error":{"message":"litellm.InternalServerError: InternalServerError: OpenAIException - 3 validation errors for ContainerListResponse\nobject\n  Field required [type=missing, input_value={'error': {'message': 'In...de': 'invalid_api_key'}}, input_type=dict]\n    For further information visit https: ...
HTTP 500

$ curl -s 'http://127.0.0.1:40540/v1/containers?model=gpt-5.6&limit=5' -H 'Authorization: Bearer $KEY_B' -w '\nHTTP %{http_code}\n' | jq -c '{object, ids: [.data[].id]}'
{"error":{"message":"litellm.InternalServerError: InternalServerError: OpenAIException - 3 validation errors for ContainerListResponse\nobject\n  Field required [type=missing, input_value={'error': {'message': 'In...de': 'invalid_api_key'}}, input_type=dict]\n    For further information visit https: ...
HTTP 500

(retrieve, files, and delete skipped: create returned no container id)

$ curl -s -X POST http://127.0.0.1:40540/v1/containers -H 'Authorization: Bearer $KEY_A' -H 'Content-Type: application/json' -d '{"name":"lit6667-before-unknown","model":"not-in-model-list"}' -w '\nHTTP %{http_code}\n'
{"error":{"message":"key not allowed to access model. This key can only access models=['gpt-5.6']. Tried to access not-in-model-list","type":"key_model_access_denied","param":"model","code":"403"}}
HTTP 403

$ curl -s -X POST http://127.0.0.1:40540/v1/containers -H 'Authorization: Bearer sk-lit6667' -H 'Content-Type: application/json' -d '{"name":"lit6667-before-nomodel"}' -w '\nHTTP %{http_code}\n'
{"error":{"message":"litellm.AuthenticationError: AuthenticationError: OpenAIException - {\n  \"error\": {\n    \"message\": \"Incorrect API key provided: None. You can find your API key at https://platform.openai.com/account/api-keys.\",\n    \"type\": \"invalid_request_error\",\n    \"param\": nul ...
HTTP 401

$ curl -s 'http://127.0.0.1:40540/v1/containers?limit=2' -H 'Authorization: Bearer sk-lit6667' -w '\nHTTP %{http_code}\n'
{"error":{"message":"litellm.InternalServerError: InternalServerError: OpenAIException - 3 validation errors for ContainerListResponse\nobject\n  Field required [type=missing, input_value={'error': {'message': 'In...de': 'invalid_api_key'}}, input_type=dict]\n    For further information visit https: ...
HTTP 500

OpenAI python SDK (openai 2.33.0) with KEY_A:
containers.create(name='sdk', extra_body={'model': 'gpt-5.6'}) -> error: AuthenticationError Error code: 401 - {'error': {'message': 'litellm.AuthenticationError: AuthenticationError: OpenAIException - {\n  "error": {\n    "message": "Incorrect API key 
containers.list(limit=3, extra_query={'model': 'gpt-5.6'}) -> error: InternalServerError Error code: 500 - {'error': {'message': "litellm.InternalServerError: InternalServerError: OpenAIException - 3 validation errors for ContainerListResponse\nobje

After (4a09c35, port 33458)

KEY_A and KEY_B are two non-admin virtual keys from POST /key/generate (models: [gpt-5.6]), each with its own user_id

$ curl -s -X POST http://127.0.0.1:33458/v1/chat/completions -H 'Authorization: Bearer $KEY_A' -H 'Content-Type: application/json' -d '{"model":"gpt-5.6","messages":[{"role":"user","content":"say hi"}],"max_tokens":5}' -w '\nHTTP %{http_code}\n' | jq -c '{model, content: .choices[0].message.content} ...
{"model":"gpt-5.6","content":""}
HTTP 200

$ curl -s -X POST http://127.0.0.1:33458/v1/containers -H 'Authorization: Bearer $KEY_A' -H 'Content-Type: application/json' -d '{"name":"lit6667-after","model":"gpt-5.6"}' -w '\nHTTP %{http_code}\n'
{"id":"cntr_bGl0ZWxsbTpjdXN0b21fbGxt...","object":"container","created_at":1788378790,"status":"running","expires_after":{"anchor":"last_active_at","minutes":20},"last_active_at":1788378790,"name":"lit6667-after"}
HTTP 200
managed id decodes to upstream id cntr_6a987ea6f9a88198b5d5753dac15a019000f734817471959

$ sleep 3  # OpenAI lists a new container about 1-2s after create (same lag when calling api.openai.com directly)

$ curl -s 'http://127.0.0.1:33458/v1/containers?model=gpt-5.6&limit=5' -H 'Authorization: Bearer $KEY_A' -w '\nHTTP %{http_code}\n' | jq -c '{object, ids: [.data[].id], has_more}'
{"object":"list","ids":["cntr_6a987ea6f9a88198b5d5753dac15a019000f734817471959"],"has_more":false}
HTTP 200
(created container cntr_6a987ea6f9a88198b5d5753dac15a019000f734817471959 present in KEY_A list: true)

$ curl -s 'http://127.0.0.1:33458/v1/containers?model=gpt-5.6&limit=5' -H 'Authorization: Bearer $KEY_B' -w '\nHTTP %{http_code}\n' | jq -c '{object, ids: [.data[].id]}'
{"object":"list","ids":[]}
HTTP 200

$ curl -s http://127.0.0.1:33458/v1/containers/$MANAGED_ID -H 'Authorization: Bearer $KEY_A' -w '\nHTTP %{http_code}\n' | jq -c '{id: .id[0:29], name, status}'
{"id":"cntr_6a987ea6f9a88198b5d5753d","name":"lit6667-after","status":"running"}
HTTP 200

$ curl -s http://127.0.0.1:33458/v1/containers/cntr_6a987ea6f9a88198b5d5753dac15a019000f734817471959 -H 'Authorization: Bearer $KEY_A' -w '\nHTTP %{http_code}\n' | jq -c '{id: .id[0:29], name, status}'
{"id":"cntr_6a987ea6f9a88198b5d5753d","name":"lit6667-after","status":"running"}
HTTP 200

$ curl -s http://127.0.0.1:33458/v1/containers/cntr_6a987ea6f9a88198b5d5753dac15a019000f734817471959 -H 'Authorization: Bearer $KEY_B' -w '\nHTTP %{http_code}\n'
{"detail":"Forbidden"}
HTTP 403

$ curl -s -X POST http://127.0.0.1:33458/v1/containers/$MANAGED_ID/files -H 'Authorization: Bearer $KEY_A' -F file=@hello.txt -w '\nHTTP %{http_code}\n' | jq -c '{id, object, path, bytes}'
{"id":"cfile_6a987eb1a8c081918f7e1673c2e9e6bf","object":"container.file","path":"/https/github.com/mnt/data/9405fdb8815879a5d615d0ec81a059b5-lit6667_hello_after.txt","bytes":25}
HTTP 200

$ curl -s http://127.0.0.1:33458/v1/containers/$MANAGED_ID/files -H 'Authorization: Bearer $KEY_A' -w '\nHTTP %{http_code}\n' | jq -c '{object, paths: [.data[].path]}'
{"object":"list","paths":["/mnt/data/9405fdb8815879a5d615d0ec81a059b5-lit6667_hello_after.txt"]}
HTTP 200

$ curl -s http://127.0.0.1:33458/v1/containers/$MANAGED_ID/files/cfile_6a987eb1a8c081918f7e1673c2e9e6bf/content -H 'Authorization: Bearer $KEY_A' -w '\nHTTP %{http_code}\n'
hello from lit6667 after

HTTP 200

$ curl -s -X DELETE http://127.0.0.1:33458/v1/containers/$MANAGED_ID -H 'Authorization: Bearer $KEY_A' -w '\nHTTP %{http_code}\n' | jq -c '{id: .id[0:29], object, deleted}'
{"id":"cntr_6a987ea6f9a88198b5d5753d","object":"container.deleted","deleted":true}
HTTP 200

$ curl -s -X POST http://127.0.0.1:33458/v1/containers -H 'Authorization: Bearer $KEY_A' -H 'Content-Type: application/json' -d '{"name":"lit6667-after-unknown","model":"not-in-model-list"}' -w '\nHTTP %{http_code}\n'
{"error":{"message":"key not allowed to access model. This key can only access models=['gpt-5.6']. Tried to access not-in-model-list","type":"key_model_access_denied","param":"model","code":"403"}}
HTTP 403

$ curl -s -X POST http://127.0.0.1:33458/v1/containers -H 'Authorization: Bearer sk-lit6667' -H 'Content-Type: application/json' -d '{"name":"lit6667-after-nomodel"}' -w '\nHTTP %{http_code}\n'
{"error":{"message":"litellm.AuthenticationError: AuthenticationError: OpenAIException - {\n  \"error\": {\n    \"message\": \"Incorrect API key provided: None. You can find your API key at https://platform.openai.com/account/api-keys.\",\n    \"type\": \"invalid_request_error\",\n    \"param\": nul ...
HTTP 401

$ curl -s 'http://127.0.0.1:33458/v1/containers?limit=2' -H 'Authorization: Bearer sk-lit6667' -w '\nHTTP %{http_code}\n'
{"error":{"message":"litellm.InternalServerError: InternalServerError: OpenAIException - 3 validation errors for ContainerListResponse\nobject\n  Field required [type=missing, input_value={'error': {'message': 'In...de': 'invalid_api_key'}}, input_type=dict]\n    For further information visit https: ...
HTTP 500

OpenAI python SDK (openai 2.33.0) with KEY_A:
containers.create(name='sdk', extra_body={'model': 'gpt-5.6'}) -> cntr_bGl0ZWxsbTpjdXN0b21fbGxtX3Byb3ZpZGVyOm9wZW5haTttb2RlbF9pZDphZTIyN2Q0MjEzNThhOGM4YWUzMDYxMjVjYjllZDcyOWQ1NzcwNjM3NjA2ZGJmOTcwM2VlNTU5YWRhYzkwMDhkO2NvbnRhaW5lcl9pZDpjbnRyXzZhOTg3ZWI4ZWQ5ODgxOTg5Y2QyOWFkNDg4MzgwMzUxMDUyNWMyMTlmMzI3N ...
  (managed id cntr_bGl0ZWxsbTpjdXN0b21fbGxt..., decodes to upstream id cntr_6a987eb8ed9881989cd29ad4883803510525c219f327495a)
sleep 3  # OpenAI lists a new container about 1-2s after create
containers.list(limit=3, extra_query={'model': 'gpt-5.6'}) -> ['cntr_6a987eb8ed9']
  (created container present in list: True )
containers.retrieve(<upstream id from list>) -> running
containers.delete(<upstream id from list>) -> None

Observations from the run:

  • OpenAI lists new containers 1-2 s late; upstream, PR untouched
  • Deleted containers linger in OpenAI's list; upstream, PR untouched
  • Retrieving a deleted container 500s on both sides; pre-existing
  • Managed-id retrieve echoes the raw id; pre-existing, more visible now
  • Unknown model on a non-admin key 403s; unchanged
  • No-model create and list still 401/500; documented Low caveat

Type

🐛 Bug Fix

Caveats (if any)

Low

  • Create or list without model still needs a global OPENAI_API_KEY
  • Create with a model_list model now returns a managed cntr_... ID instead of the raw upstream one, matching the Responses API. Every container endpoint accepts both forms
  • Model-routed create goes through router retries like /v1/responses and /v1/files do; num_retries: 0 turns that off
  • List ignores limit and other pagination params; pre-existing, untouched here

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/25f80f570f9b43fe82383d92e8dacdbf
Open in Devin Desktop: https://app.devin.ai/desktop/session/25f80f570f9b43fe82383d92e8dacdbf?variant=devin


Note

Medium Risk
Changes proxy credential routing and container ID shape for model-routed creates; behavior is covered by new tests and mirrors existing files/responses routing, but touches auth-sensitive gateway paths.

Overview
Fixes proxy Containers API auth and routing when OpenAI credentials live only in model_list, not in a global OPENAI_API_KEY.

Router: _init_containers_api_endpoints now sends create/list (no managed cntr_... ID) through _ageneric_api_call_with_fallbacks when the caller passes a non-empty model, injecting that deployment’s api_key / api_base. Unknown model names use passthrough_on_no_deployment=True so the handler is still invoked directly without a 400, same as before.

Proxy: GET /v1/containers forwards model from the query string into the request payload so list can pick a deployment.

SDK: create_container applies managed-ID encoding after async upstream responses via _encode_created_container_id, so routed creates return embeddable cntr_... IDs for retrieve/delete/files instead of raw upstream IDs.

Reviewed by Cursor Bugbot for commit 4a09c35. Bugbot is set up for automated code reviews on this repo. Configure here.

…ments

Container create and list requests had no container ID to decode, so the
router called the provider handler directly and the OpenAI transformation
fell back to the global OPENAI_API_KEY. Proxies configured only with
model_list credentials sent Authorization: Bearer None. Route through
_ageneric_api_call_with_fallbacks when the caller passes a model, expose
the list endpoint's model query param to the router, and encode the
managed container ID on the async create path so follow-up calls route
to the same deployment.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

No action taken: PR #39220 (BerriAI/litellm, author devin-ai-integration[bot]) has no labels and lacks the enterprise label, so it is out of scope. No risk label applied, no Linear routing performed.

@codspeed

codspeed Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_containers_route_model_list_creds (4a09c35) with litellm_internal_staging (a677242)

Open in CodSpeed

@greptile-apps

greptile-apps Bot commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR routes container creation and listing through configured model deployments and preserves deployment metadata in IDs returned by asynchronous creation

  • Forwards the list endpoint’s model query parameter into shared request processing
  • Uses deployment credentials for model-routed container create and list calls
  • Encodes asynchronous create responses for later container operations

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename Overview
litellm/containers/main.py Adds post-await managed-ID encoding for asynchronous container creation
litellm/proxy/container_endpoints/endpoints.py Passes the list endpoint’s model query parameter into request processing
litellm/router.py Routes model-bearing container create and list operations through configured deployments
tests/router_unit_tests/test_router_endpoints.py Adds coverage for configured, absent, and unknown deployment routing
tests/test_litellm/containers/test_container_api.py Adds functional coverage for managed-ID encoding after asynchronous creation

Reviews (4): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile

Comment thread litellm/router.py
Comment thread tests/test_litellm/containers/test_container_api.py Outdated
@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.00000% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
litellm/router.py 66.66% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

…eployment

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri

Copy link
Copy Markdown
Contributor

bugbot run

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 4a09c35. Configure here.

@mateo-berri

Copy link
Copy Markdown
Contributor

@greptileai

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mateo-berri
mateo-berri merged commit afb4d76 into litellm_internal_staging Sep 2, 2026
83 of 84 checks passed
@mateo-berri
mateo-berri deleted the litellm_containers_route_model_list_creds branch September 2, 2026 20:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants