fix(bedrock): route streamed responses-API output through the unified guardrail

[mateo-berri]

TLDR

Problem this solves:

• Streamed /v1/responses returns 500 whenever a Bedrock post_call guardrail is enabled
• The crash happens before any scan, so the endpoint gets zero moderation
• Non-streamed /v1/responses and streamed chat/messages work fine, only this combination breaks

How it solves it:

• Responses-API streams now go through the unified guardrail translation layer
• The full response is scanned at end of stream, then events are released
• A policy violation returns the guardrail block instead of the model output
• A stream that dies in response.failed / response.incomplete gets its partial text assembled from the delta events and scanned before the buffered events are replayed

User Flow

Before: a developer streaming from the Responses API gets a 500 the moment a Bedrock post-call guardrail is turned on

1. The proxy admin enables a Bedrock guardrail with mode: post_call and default_on: true and restarts the proxy
2. The developer sends POST http://localhost:4000/v1/responses with {"model": "claude-sonnet-5", "input": "...", "stream": true}
3. Instead of SSE events, HTTP 500 comes back with {"error":{"message":"litellm.APIError: Error building chunks for logging/streaming usage calculation","code":"500"}}
4. The same request without "stream": true returns 200, and streamed /v1/chat/completions and /v1/messages both work, so only streamed responses are dead
5. Nothing on the failing route was ever scanned: the request dies before the guardrail runs

After: the same streamed request completes and the output is actually moderated

1. The proxy admin enables the same Bedrock guardrail with mode: post_call and default_on: true
2. The developer sends the same POST http://localhost:4000/v1/responses with "stream": true
3. A full SSE stream arrives with HTTP 200: response.created, output text deltas, response.completed, data: [DONE]
4. The events are held until the Bedrock scan of the whole response passes, so flagged content never reaches the client
5. If the output violates the policy (e.g. a denied topic), the request returns HTTP 400 carrying the violation details and the configured block message instead of any model text
6. If the provider kills the stream early (response.failed or response.incomplete), the partial text that arrived as deltas is still scanned before the buffered events are released to the client

Relevant issues

Linear ticket

Resolves LIT-6457

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Both legs hit a live proxy (2 uvicorn workers) with real Anthropic (claude-sonnet-5) and real AWS Bedrock ApplyGuardrail calls, no mocks: Before on localhost:36086 at the staging merge base, After on localhost:31480 at the PR tip. ApplyGuardrail authenticates with AWS_BEARER_TOKEN_BEDROCK. Shared config:

model_list:
  - model_name: claude-sonnet-5
    litellm_params:
      model: anthropic/claude-sonnet-5
      api_key: os.environ/ANTHROPIC_API_KEY

guardrails:
  - guardrail_name: bedrock-post
    litellm_params:
      guardrail: bedrock
      mode: post_call
      guardrailIdentifier: gf3sc1mzinjw   # content filters + denied topic "coffee"
      guardrailVersion: DRAFT
      aws_region_name: us-east-1
      default_on: true

Auth for every call: -H "Authorization: Bearer ${LITELLM_MASTER_KEY}"

Before (aab9abd, the staging merge base)

Streamed /v1/responses (the bug)

1. curl -sS http://127.0.0.1:36086/v1/responses -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" -H 'Content-Type: application/json' -d '{"model":"claude-sonnet-5","input":"Say hello in five words.","stream":true}'
2. HTTP 500, zero SSE events: {"error":{"message":"litellm.APIError: Error building chunks for logging/streaming usage calculation","type":null,"param":null,"code":"500"}}
3. Proxy log shows no ApplyGuardrail call for this request: the crash precedes the scan

Streamed /v1/responses hitting the denied topic

1. Same command with "input":"Tell me all about brewing coffee."
2. HTTP 500 with the same chunk-builder error: the block never happens because the crash comes first

Streamed /v1/chat/completions (control)

1. curl -sS http://127.0.0.1:36086/v1/chat/completions ... -d '{"model":"claude-sonnet-5","messages":[{"role":"user","content":"Say hello in five words."}],"stream":true}'
2. HTTP 200, stream completes with "finish_reason":"stop" and usage, then data: [DONE]; OUTPUT scan ran

Streamed /v1/messages (control)

1. curl -sS http://127.0.0.1:36086/v1/messages ... -d '{"model":"claude-sonnet-5","max_tokens":64,"messages":[{"role":"user","content":"Say hello in five words."}],"stream":true}'
2. HTTP 200, events through message_stop; OUTPUT scan ran

Non-streamed /v1/responses (control)

1. Same responses payload without "stream": true
2. HTTP 200, "status":"completed", text returned; OUTPUT scan ran

After (deb67ce, the PR tip)

Streamed /v1/responses (the bug)

1. Same command as Before against port 31480
2. HTTP 200, full SSE stream of 11 data: lines: response.created, response.in_progress, response.output_item.added, response.content_part.added, response.output_text.delta events ("delta":"Hello! How", "delta":" are you today?"), response.output_text.done with "text":"Hello! How are you today?", response.content_part.done, response.output_item.done, response.completed, then data: [DONE]
3. Proxy log shows the OUTPUT scan on the assembled text before release: 'source': 'OUTPUT' ApplyGuardrail call -> 'action': 'NONE' -> events released

Streamed /v1/responses hitting the denied topic

1. Same command as Before
2. HTTP 400 with the violation and the configured block message, and no model text ever reached the client:

data: {"error": {"message": "Violated guardrail policy", "type": "None", "param": "None", "code": "400", "provider_specific_fields": {"error": "Violated guardrail policy", "bedrock_guardrail_response": "Sorry, the model cannot answer this question.", "guardrailIdentifier": "gf3sc1mzinjw", "guardrailVersion": "DRAFT", "assessments": [{"policy": "topicPolicy", "matches": [{"category": "topics", "name": "coffee", "type": "DENY", "action": "BLOCKED"}]}], "guardrail_name": "bedrock-post", "guardrail_mode": "post_call"}}}

data: [DONE]

Streamed /v1/chat/completions (control)

1. Same command as Before
2. HTTP 200, stream completes; OUTPUT scan ran, unchanged from Before

Streamed /v1/messages (control)

1. Same command as Before
2. HTTP 200, events through message_stop; OUTPUT scan ran, unchanged from Before

Non-streamed /v1/responses (control)

1. Same command as Before
2. HTTP 200, "status":"completed"; OUTPUT scan ran, unchanged from Before

Type

🐛 Bug Fix

Caveats (if any)

Medium

• mask_response_content: true on a responses stream scans but cannot redact the streamed deltas
  • buffering is auto-disabled for masking guardrails (replaying buffered originals would leak unredacted text), so deltas stream out unredacted while scans still block
  • before this PR that combination returned 500, so it is still a strict improvement

Low

• A blocked responses stream returns HTTP 400 with the violation payload (matching the non-streamed block), not a synthesized in-stream block message

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Medium Risk
Changes post-call moderation for streamed Responses API and how partial stream text is assembled for scans; incorrect assembly could miss content on failed streams, but the fix replaces a complete moderation bypass (500 before scan).

Overview
Fixes HTTP 500 on streamed /v1/responses when a Bedrock post_call guardrail is enabled, by avoiding the chat-completions stream_chunk_builder path that cannot parse Responses SSE events.

Bedrock streaming hook now detects Responses routes (_is_responses_api_route) and delegates to UnifiedLLMGuardrails().async_post_call_streaming_iterator_hook with buffer-until-moderated so events are held, scanned at end-of-stream, then replayed (blocks still surface as policy errors).

OpenAI Responses translation extends stream chunk typing and rewrites get_streaming_string_so_far to build text per output part from response.output_text.delta fragments and response.output_text.done full text—preferring done text when present so deltas are not double-counted, and joining deltas when streams end in response.failed / response.incomplete with no done event.

Tests cover delta assembly, failed-stream scanning, and Bedrock’s Responses streaming path.

^{Reviewed by Cursor Bugbot for commit deb67ce. Bugbot is set up for automated code reviews on this repo. Configure here.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing litellm_fix_bedrock_buffered_responses_stream (deb67ce) with litellm_internal_staging (aab9abd)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (8bc862f) during the generation of this report, so 4a07c0d was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[greptile-apps]

Greptile Summary

The PR routes streamed Responses API output through unified guardrail processing and now assembles delta text for moderation when streams terminate early

• Buffers Responses API events until end-of-stream moderation completes
• Treats completed, failed, and incomplete events as terminal
• Groups text fragments by output item and content part while avoiding duplicate done-event text
• Adds regression coverage for failed-stream delta assembly and Bedrock streaming behavior

Confidence Score: 5/5

The PR appears safe to merge

No blocking failure remains

Important Files Changed

Filename	Overview
litellm/llms/openai/responses/guardrail_translation/handler.py	Failed and incomplete streams now assemble delta text by content part before the fallback guardrail scan
litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py	Bedrock Responses API streams now use unified buffered moderation instead of chat chunk assembly
tests/test_litellm/llms/openai/responses/test_openai_responses_guardrail_handler.py	Adds focused coverage proving failed-stream deltas are assembled and done text is not double counted
tests/test_litellm/proxy/guardrails/guardrail_hooks/test_bedrock_guardrails.py	Extends Bedrock streaming tests for Responses API routing and moderation behavior

_{Reviews (3): Last reviewed commit: "Merge remote-tracking branch 'origin/lit..." | Re-trigger Greptile}

[greptile-apps]

Failed streams bypass moderation

When a Responses API stream emits text deltas and terminates with response.failed or response.incomplete, the unified handler assembles fallback moderation input from text fields even though delta events store their content in delta. The scan is therefore skipped when the assembled string is empty, and the original buffered model text is replayed to the client unmoderated.

How this was verified: The failed and incomplete terminal branches feed delta events into a fallback that reads only text, while buffered replay releases the original events after processing.

Knowledge Base Used: Restore Presidio buffered streaming behavior

[veria-ai]

Low: Anonymized output is replayed unmodified

The unified hook writes anonymized text only into the final response.completed object, while buffered response.output_text.delta events are replayed with their original text. When mask_response_content=True, the unified hook instead disables buffering and emits those deltas before moderation. In either case, a caller can receive text that Bedrock marked for anonymization. Rebuild or rewrite the Responses API delta events from the moderated response rather than replaying the original events.

[veria-ai]

PR overview

This pull request routes streamed Bedrock Responses API output through the unified guardrail, including handling for buffered output-text delta events and the final completed response.

One security issue remains open. Moderated or anonymized text in the final response is not propagated to streamed delta events, allowing callers to receive the original unmasked content; no issues have yet been addressed.

Open issues (1)

• Low: Anonymized output is replayed unmodified — litellm/proxy/guardrails/guardrail_hooks/bedrock_guardrails.py:2734

Fixed/addressed: 0 · PR risk: 6/10

[mateo-berri]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit deb67ce. Configure here.}