Skip to content

fix(bedrock): honor BEDROCK_MANTLE_API_BASE on bedrock/mantle messages and chat URLs - #39364

Merged
mateo-berri merged 2 commits into
litellm_internal_stagingfrom
litellm_fix_bedrock_mantle_messages_env_api_base
Sep 2, 2026
Merged

mateo-berri merged 2 commits into
litellm_internal_stagingfrom
litellm_fix_bedrock_mantle_messages_env_api_base

Conversation

@mateo-berri

@mateo-berri mateo-berri commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • bedrock/mantle/* deployments ignored BEDROCK_MANTLE_API_BASE and always called the public bedrock-mantle.<region>.api.aws host
  • Both the /v1/messages route and /v1/chat/completions on those models were affected, since they share one URL builder
  • Operators had to repeat api_base on every deployment instead of setting the documented env var once

How it solves it:

  • build_mantle_messages_url now falls back to BEDROCK_MANTLE_API_BASE after api_base and aws_bedrock_runtime_endpoint
  • Explicit per-deployment overrides keep winning over the env var
  • The env var is shared with the OpenAI-surface bedrock_mantle/* routes, which need it to end in /v1 or /openai/v1 for plain chat models, so that suffix is dropped before /anthropic/v1/messages is appended and one value serves every Mantle route
  • The bedrock_mantle/* provider already honored the env var, so the two Mantle spellings now agree

User Flow

Before: an operator whose Mantle endpoint is a private host sets BEDROCK_MANTLE_API_BASE=https://bedrock-mantle.us-east-1.api.aws.corp.example, adds a bedrock/mantle/anthropic.claude-haiku-4-5 deployment with no api_base, and every call still goes to the public AWS host their network cannot reach

  1. They set BEDROCK_MANTLE_API_BASE=https://bedrock-mantle.us-east-1.api.aws.corp.example and start the proxy with a bedrock/mantle/anthropic.claude-haiku-4-5 deployment named mantle-claude
  2. They send POST https://litellm-domain/v1/messages with {"model": "mantle-claude", "max_tokens": 32, "messages": [{"role": "user", "content": "Say pong"}]}
  3. The call fails with a 500 connection timeout naming bedrock-mantle.us-east-1.api.aws, and their private endpoint's access log shows no request
  4. They send POST https://litellm-domain/v1/chat/completions with the same deployment and get the same timeout, since that model is served through the Anthropic messages URL
  5. Only after they copy the private host into api_base on the deployment do POST https://litellm-domain/v1/messages and POST https://litellm-domain/v1/chat/completions reach their private host

After: the env var alone is enough, and both routes come back 200 through the private host

  1. They set BEDROCK_MANTLE_API_BASE=https://bedrock-mantle.us-east-1.api.aws.corp.example and start the proxy with a bedrock/mantle/anthropic.claude-haiku-4-5 deployment named mantle-claude
  2. They send POST https://litellm-domain/v1/messages with {"model": "mantle-claude", "max_tokens": 32, "messages": [{"role": "user", "content": "Say pong"}]}
  3. The call returns 200 with "type": "message" and "text": "pong", and their private endpoint's access log shows POST /anthropic/v1/messages
  4. They send POST https://litellm-domain/v1/chat/completions with the same deployment and get 200 with "content": "Pong", again through their private host
  5. A deployment that already carries api_base keeps using that value, so POST https://litellm-domain/v1/messages against it behaves exactly as before

Relevant issues

Linear ticket

Resolves LIT-6641

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
  • My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup, identical on both sides: a local TLS reverse proxy plays the customer-shaped private host https://bedrock-mantle.us-east-1.api.aws.localtest.me:9443 (resolving to 127.0.0.1) and forwards to the real public Mantle host, appending one line per hit to its access log. Real calls, real spend, AWS_BEARER_TOKEN_BEDROCK from the CI account. Both proxies run with no DB, 2 uvicorn workers each (--num_workers 2), on a random port with this env and config. The Before proxy runs the merge base on port 44776 and the After proxy runs the PR tip on port 42383, which is why the curls below name those ports instead of 4000

BEDROCK_MANTLE_API_BASE=https://bedrock-mantle.us-east-1.api.aws.localtest.me:9443
SSL_CERT_FILE=<certifi bundle plus the reverse proxy CA>
AWS_BEARER_TOKEN_BEDROCK=<real token>
model_list:
  - model_name: mantle-claude
    litellm_params:
      model: bedrock/mantle/anthropic.claude-haiku-4-5
      aws_region_name: us-east-1
  - model_name: mantle-claude-apibase
    litellm_params:
      model: bedrock/mantle/anthropic.claude-haiku-4-5
      aws_region_name: us-east-1
      api_base: https://bedrock-mantle.us-east-1.api.aws.localtest.me:9443
  - model_name: mantle-oss
    litellm_params:
      model: bedrock_mantle/openai.gpt-oss-120b
      aws_region_name: us-east-1
general_settings:
  master_key: sk-1234

Each case shows the curl, the response, and the new lines in the private host's access log, which is the operator's own evidence of whether the request reached their endpoint. Cases 1, 2 and 4 are the env-var-only deployment on the messages, chat completions and responses routes; case 3 is the control with api_base on the deployment, which must not change. Cases 5 and 6 rerun both proxies with the same env except BEDROCK_MANTLE_API_BASE=https://bedrock-mantle.us-east-1.api.aws.localtest.me:9443/v1, the shape plain chat models on bedrock_mantle/* need: case 5 is the messages route again and case 6 is the existing gpt-oss chat route that must keep working with that value

Before (ffc0a8e)

Case 1: Anthropic messages with BEDROCK_MANTLE_API_BASE only (mantle-claude)

  1. Run
    curl -sD - http://127.0.0.1:44776/v1/messages -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-claude","max_tokens":32,"messages":[{"role":"user","content":"Say pong"}]}'
    
  2. Response
    HTTP/1.1 200 OK
    body: {"model":"mantle-claude","id":"msg_bdrk_gjh3yu6ex7etgvz3lcdcwuyuc26khkxjsrjkcnoxkwdy4uou5bwq","type":"message","role":"assistant","content":[{"type":"text","text":"Pong"}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens
    
  3. New lines in the private host's access log
    (none)
    

Case 2: Chat completions with BEDROCK_MANTLE_API_BASE only (mantle-claude)

  1. Run
    curl -sD - http://127.0.0.1:44776/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-claude","messages":[{"role":"user","content":"Say pong"}],"max_tokens":32}'
    
  2. Response
    HTTP/1.1 200 OK
    body: {"id":"chatcmpl-8d039237-0743-4d4b-ba6a-360dad2f9ed4","created":1788373249,"model":"mantle-claude","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"pong","role":"assistant","provider_specific_fields":{"citations":nu
    
  3. New lines in the private host's access log
    (none)
    

Case 3: Anthropic messages with api_base on the deployment, control (mantle-claude-apibase)

  1. Run
    curl -sD - http://127.0.0.1:44776/v1/messages -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-claude-apibase","max_tokens":32,"messages":[{"role":"user","content":"Say pong"}]}'
    
  2. Response
    HTTP/1.1 200 OK
    body: {"model":"mantle-claude-apibase","id":"msg_bdrk_6g3wubemjqivdhjwe56dcpl4en4dbjutwp2zgzy7q3knf7hap3lq","type":"message","role":"assistant","content":[{"type":"text","text":"pong"}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"inpu
    
  3. New lines in the private host's access log
    POST client_sni=bedrock-mantle.us-east-1.api.aws.localtest.me upstream=('bedrock-mantle.us-east-1.api.aws', 443) path=/anthropic/v1/messages status=200
    

Case 4: Responses with BEDROCK_MANTLE_API_BASE only, chat bridge (mantle-claude)

  1. Run
    curl -sD - http://127.0.0.1:44776/v1/responses -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-claude","input":"Say pong","max_output_tokens":32}'
    
  2. Response
    HTTP/1.1 200 OK
    body: {"id":"resp_zhgw4sTXfnHjeRY-olKLEcepE8okb_9-oXff7q5TDorYFnh8a7ORWozk5BQftb8URPPdhmP_8glBHzYwXXkEogLa48pz6vAM0dkjyY39iguWiUbbFDeFmwIa2Lq_TI990ko0sj5WixSTYQeml2rsOpv-8yQ0bmngkjiu0FJHEsPm5vs4wjrCm0zkQ96y3SDI63bFyTrXnpisSLdEWodH7mJ-_Trxoyo_UxZZ0rgHFdomGOzzBVCAqcLA
    
  3. New lines in the private host's access log
    (none)
    

Case 5: Anthropic messages with BEDROCK_MANTLE_API_BASE ending in /v1 (mantle-claude)

  1. Run
    curl -sD - http://127.0.0.1:44776/v1/messages -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-claude","max_tokens":32,"messages":[{"role":"user","content":"Say pong"}]}'
    
  2. Response
    HTTP/1.1 200 OK
    body: {"model":"mantle-claude","id":"msg_bdrk_cfszohn2c7h4xgfpz5rzoknqg2uvf2tc6jo3q7g6o6kgnrzd5faa","type":"message","role":"assistant","content":[{"type":"text","text":"pong"}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens
    
  3. New lines in the private host's access log
    (none)
    

Case 6: Plain chat completions on gpt-oss with BEDROCK_MANTLE_API_BASE ending in /v1, control (mantle-oss)

  1. Run
    curl -sD - http://127.0.0.1:44776/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-oss","messages":[{"role":"user","content":"Say pong"}],"max_tokens":64}'
    
  2. Response
    HTTP/1.1 200 OK
    x-litellm-model-api-base: https://bedrock-mantle.us-east-1.api.aws.localtest.me:9443/v1
    body: {"id":"chatcmpl-0557e7e2-b411-41a2-bb92-f41a2ba92387","created":1788373903,"model":"mantle-oss","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"pong","role":"assistant","reasoning":"The user says: \"Say pong\". Pro
    
  3. New lines in the private host's access log
    POST client_sni=bedrock-mantle.us-east-1.api.aws.localtest.me upstream=('bedrock-mantle.us-east-1.api.aws', 443) path=/v1/chat/completions status=200
    

After (49c69c4)

Case 1: Anthropic messages with BEDROCK_MANTLE_API_BASE only (mantle-claude)

  1. Run
    curl -sD - http://127.0.0.1:42383/v1/messages -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-claude","max_tokens":32,"messages":[{"role":"user","content":"Say pong"}]}'
    
  2. Response
    HTTP/1.1 200 OK
    body: {"model":"mantle-claude","id":"msg_bdrk_zdj6cjrugmnk2357iffzkp5tryel3zvasitbcmyuqc27usxsqbpq","type":"message","role":"assistant","content":[{"type":"text","text":"pong"}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens
    
  3. New lines in the private host's access log
    POST client_sni=bedrock-mantle.us-east-1.api.aws.localtest.me upstream=('bedrock-mantle.us-east-1.api.aws', 443) path=/anthropic/v1/messages status=200
    

Case 2: Chat completions with BEDROCK_MANTLE_API_BASE only (mantle-claude)

  1. Run
    curl -sD - http://127.0.0.1:42383/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-claude","messages":[{"role":"user","content":"Say pong"}],"max_tokens":32}'
    
  2. Response
    HTTP/1.1 200 OK
    body: {"id":"chatcmpl-43799335-879b-4dac-be63-100056b867ed","created":1788375275,"model":"mantle-claude","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"pong","role":"assistant","provider_specific_fields":{"citations":nu
    
  3. New lines in the private host's access log
    POST client_sni=bedrock-mantle.us-east-1.api.aws.localtest.me upstream=('bedrock-mantle.us-east-1.api.aws', 443) path=/anthropic/v1/messages status=200
    

Case 3: Anthropic messages with api_base on the deployment, control (mantle-claude-apibase)

  1. Run
    curl -sD - http://127.0.0.1:42383/v1/messages -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-claude-apibase","max_tokens":32,"messages":[{"role":"user","content":"Say pong"}]}'
    
  2. Response
    HTTP/1.1 200 OK
    body: {"model":"mantle-claude-apibase","id":"msg_bdrk_2jm3rex6fsgbubbmivs5w5xd4dwtx23xwu5qjkmmt27nbfds63sa","type":"message","role":"assistant","content":[{"type":"text","text":"pong"}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"inpu
    
  3. New lines in the private host's access log
    POST client_sni=bedrock-mantle.us-east-1.api.aws.localtest.me upstream=('bedrock-mantle.us-east-1.api.aws', 443) path=/anthropic/v1/messages status=200
    

Case 4: Responses with BEDROCK_MANTLE_API_BASE only, chat bridge (mantle-claude)

  1. Run
    curl -sD - http://127.0.0.1:42383/v1/responses -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-claude","input":"Say pong","max_output_tokens":32}'
    
  2. Response
    HTTP/1.1 200 OK
    body: {"id":"resp_JPUVLkrKPAeDkGTRjSn4u4TXNs-QbKWlCpYOsxe7GUQhctRubvJAx2imIRPXo7hoL78RIRGUAjdoeAtzhhWdzCy7XzgcJYZaPz0KIEZdJeKF3tVsBkVHrhaYxfuAq-dBcO5gf-ZoyozHwLTwBc-D5gz0jdVRh-0vF48y1OHfSkg7taYtLjyAZQPec02jCpR4ylUw3CQ5oA4klNw21Cpnc166gkmdPWIvrjGz_wHYltgAjnDGaOaEEQs8
    
  3. New lines in the private host's access log
    POST client_sni=bedrock-mantle.us-east-1.api.aws.localtest.me upstream=('bedrock-mantle.us-east-1.api.aws', 443) path=/anthropic/v1/messages status=200
    

Case 5: Anthropic messages with BEDROCK_MANTLE_API_BASE ending in /v1 (mantle-claude)

  1. Run
    curl -sD - http://127.0.0.1:42383/v1/messages -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-claude","max_tokens":32,"messages":[{"role":"user","content":"Say pong"}]}'
    
  2. Response
    HTTP/1.1 200 OK
    body: {"model":"mantle-claude","id":"msg_bdrk_hzkwmdwotqrt47wllwa3mthgv2stpcjsrmllnyv7h6itpgx2qutq","type":"message","role":"assistant","content":[{"type":"text","text":"pong"}],"stop_reason":"end_turn","stop_sequence":null,"stop_details":null,"usage":{"input_tokens
    
  3. New lines in the private host's access log
    POST client_sni=bedrock-mantle.us-east-1.api.aws.localtest.me upstream=('bedrock-mantle.us-east-1.api.aws', 443) path=/anthropic/v1/messages status=200
    

Case 6: Plain chat completions on gpt-oss with BEDROCK_MANTLE_API_BASE ending in /v1, control (mantle-oss)

  1. Run
    curl -sD - http://127.0.0.1:42383/v1/chat/completions -H 'Authorization: Bearer sk-1234' -H 'Content-Type: application/json' -d '{"model":"mantle-oss","messages":[{"role":"user","content":"Say pong"}],"max_tokens":64}'
    
  2. Response
    HTTP/1.1 200 OK
    x-litellm-model-api-base: https://bedrock-mantle.us-east-1.api.aws.localtest.me:9443/v1
    body: {"id":"chatcmpl-1c9b3b2a-a669-418b-b16e-aa8dc9afd889","created":1788375456,"model":"mantle-oss","object":"chat.completion","choices":[{"finish_reason":"stop","index":0,"message":{"content":"pong","role":"assistant","reasoning":"The user says \"Say pong\". So w
    
  3. New lines in the private host's access log
    POST client_sni=bedrock-mantle.us-east-1.api.aws.localtest.me upstream=('bedrock-mantle.us-east-1.api.aws', 443) path=/v1/chat/completions status=200
    

Observations from the run, none caused by this PR:

  • Before, cases 1, 2, 4 and 5 still return 200 because this machine can reach the public host; the operator's private endpoint sees nothing

  • After, cases 1, 2, 4 and 5 hit the private host with POST /anthropic/v1/messages and return the same bodies

  • Case 3 hits the private host on both sides, so api_base on the deployment is unchanged

  • Case 6 hits the private host with POST /v1/chat/completions on both sides, so the /v1 env value keeps serving plain chat

  • Not verified live: SigV4 signing against the private host (the QA account authenticates with a bearer token) and the aws_bedrock_runtime_endpoint override, both covered by the unit tests only

Type

🐛 Bug Fix

Caveats

Low: only a trailing /v1 or /openai/v1 is dropped from BEDROCK_MANTLE_API_BASE before the messages path is appended. A value that ends in some other path keeps it, so a private gateway that mounts Mantle under a different prefix needs api_base on the bedrock/mantle/* deployment, which keeps winning over the env var

Final Attestation

  • I have followed the rules in this template, including the ones inside HTML comments, and I have not fabricated any proof or content

@codspeed

codspeed Bot commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_fix_bedrock_mantle_messages_env_api_base (49c69c4) with litellm_internal_staging (b600f02)

Open in CodSpeed

@greptile-apps

greptile-apps Bot commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR makes Bedrock Mantle messages and chat requests honor BEDROCK_MANTLE_API_BASE while preserving explicit deployment endpoint precedence.

  • Adds the environment variable as the final endpoint override fallback.
  • Normalizes OpenAI-style /v1 and /openai/v1 suffixes before appending the Anthropic messages path.
  • Adds coverage for environment-based endpoints and explicit override precedence across both Mantle configurations.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
litellm/llms/bedrock/common_utils.py Adds normalized environment endpoint fallback while retaining existing explicit override precedence.
tests/test_litellm/llms/bedrock/test_mantle.py Covers supported environment endpoint forms and verifies explicit endpoint overrides continue to win.

Reviews (2): Last reviewed commit: "fix(bedrock): drop the OpenAI base suffi..." | Re-trigger Greptile

Comment thread litellm/llms/bedrock/common_utils.py Outdated
Comment thread tests/test_litellm/llms/bedrock/test_mantle.py Outdated
@codecov

codecov Bot commented Sep 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@mateo-berri

Copy link
Copy Markdown
Contributor Author

bugbot run

@mateo-berri

Copy link
Copy Markdown
Contributor Author

@greptileai

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 49c69c4. Configure here.

@tin-berri tin-berri left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mateo-berri
mateo-berri merged commit 4286be8 into litellm_internal_staging Sep 2, 2026
82 of 83 checks passed
@mateo-berri
mateo-berri deleted the litellm_fix_bedrock_mantle_messages_env_api_base branch September 2, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants