fix(streaming): keep response id stable across streamed chunks

[Timik232]

TLDR

Problem this solves:

• Any provider streaming through the generic-chunk path without an upstream response id (GigaChat today) emitted a fresh top-level id on every streamed chunk
• Clients that merge deltas by chunk id (e.g. goose) split one reply into many separate messages

How it solves it:

• The stream wrapper now pins one response id per streamed response from the first chunk it emits, mirroring the existing created pinning

User Flow

Before: a developer using goose behind the LiteLLM proxy with a model whose provider sends no response ids sees one answer arrive as many separate messages

1. The developer points goose at the proxy's OpenAI-compatible endpoint with their key
2. goose sends POST /v1/chat/completions with "stream": true for such a model (e.g. gigachat/GigaChat-2-Max)
3. Each SSE chunk comes back with a different "id": "chatcmpl-<uuid>" (6 chunks -> 6 different ids)
4. goose merges streamed deltas by chunk id, so the session ends up with 6 separate assistant messages instead of one, and the chat shows the answer shattered into pieces

After: the same request streams with one stable id, so goose renders one message

1. The developer points goose at the proxy's OpenAI-compatible endpoint with their key
2. goose sends the same POST /v1/chat/completions with "stream": true for the same model
3. Every SSE chunk now carries the same "id" (6 chunks -> 1 id)
4. goose renders the answer as a single assistant message

Relevant issues

Fixes #38098

Related, same bug class fixed elsewhere: #32607 (responses bridge, fixed by #32610), #36700 (ollama)

Linear ticket

Resolves LIT-6614

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/litellm_core_utils/test_streaming_handler.py -v (132/132 after merging the latest base branch in)
• My PR passes all required CI/CD checks (e.g. lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

The code path is provider-agnostic (any GenericStreamingChunk-based provider). Fresh QA below reruns the proof at the merge base and at the merged PR tip using bedrock/invoke/meta.llama3-8b-instruct-v1:0 (real AWS Bedrock credentials, real billable calls) since it exercises the same generic-chunk path GigaChat does; goose 1.48.0 is driven as a real interactive TUI under tmux, the actual client from #38098.

Before (merge base 77813ae)

Streamed chunks through the proxy

curl -sN .../v1/chat/completions -d '{"model":"llama-native","stream":true,"messages":[{"role":"user","content":"Write three short facts about space"}],"max_tokens":120}' | grep -o '"id":"chatcmpl-[a-zA-Z0-9-]*"' | sort | uniq -c

Observed: 121 chunks, 121 unique ids, confirmed on two runs.

Client (goose) receives the stream

goose TUI rendered the reply as what looked like one flowing block, but its wire log (llm_request.0.jsonl) showed 166 chunks carrying an id, 166 unique ids, and goose's own SQLite session store persisted 166 separate one-token assistant message rows instead of one merged message, worse than the PR's own 6-chunk example. /v1/responses and /v1/messages streaming were unaffected either way (already fixed by #32610 / never exposed to the bug).

After (PR tip 24393be)

Streamed chunks through the proxy

Same curl against a proxy on this tip: 121 chunks, 1 unique id, all sharing chatcmpl-d41c6992-8d6c-4dc7-9ef8-e15770f93a49.

Client (goose) receives the stream

Same goose TUI flow: wire log shows 114 chunks sharing 1 id; goose renders (and persists) one coherent assistant message. /v1/responses (response.id, item_id) and /v1/messages (message.id) streaming remain stable, no regression.

Type

🐛 Bug Fix

Caveats (if any)

Low

• Providers that supply their own chunk ids are unaffected: a provider-supplied id still wins (pinned earlier on the OpenAI-chunk path)
• Only consumers relying on a nonstandard fresh id per chunk will see a behavior change
• OCI's generic and Cohere streaming (litellm/llms/oci/chat/generic.py, cohere.py) builds ModelResponseStream directly in a fully-overridden chunk_creator, bypassing this fix entirely; it keeps emitting a fresh id per chunk, same defect class, pre-existing and out of this PR's scope

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

---

Note

Low Risk
Small, localized change to stream metadata assembly; provider-supplied ids are unchanged and behavior only affects consumers that depended on per-chunk random ids.

Overview
Fixes streaming responses where every chunk got a new id when providers use the generic-chunk path (e.g. GigaChat) and do not send an upstream response id. Clients that merge deltas by chunk id (such as goose) were treating one reply as multiple assistant messages.

CustomStreamWrapper.model_response_creator now pins response_id from the first emitted chunk when the wrapper does not already have one—same pattern as existing created pinning (#11437). If self.response_id is already set (including from the provider/OpenAI path via set_model_id), that value still wins on every chunk.

Adds TestStableStreamingResponseId covering shared ids across generic chunks, pinning on first model_response_creator(), and provider-supplied ids.

^{Reviewed by Cursor Bugbot for commit 24393be. Bugbot is set up for automated code reviews on this repo. Configure here.}

• 24393be passes /live-pr-risk

[CLAassistant]

All committers have signed the CLA.

[greptile-apps]

Greptile Summary

The PR keeps the generated response ID stable across chunks emitted by the generic streaming path.

• Caches the first generated ModelResponseStream ID on the stream wrapper.
• Adds coverage for generic streams, repeated response creation, and provider-supplied IDs.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
litellm/litellm_core_utils/streaming_handler.py	Pins the first generated response ID for reuse across streamed chunks; the previously flagged redundant comments are no longer present.
tests/test_litellm/litellm_core_utils/test_streaming_handler.py	Adds focused regression tests confirming stable generated IDs and preservation of explicitly supplied IDs.

_{Reviews (3): Last reviewed commit: "Merge branch 'litellm_internal_staging' ..." | Re-trigger Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[codspeed]

Merging this PR will not alter performance

✅ 31 untouched benchmarks

---

_{Comparing Timik232:bugfix/streaming-stable-response-id (24393be) with litellm_internal_staging (82dd36c)¹}

Footnotes

1. No successful run was found on litellm_internal_staging (caeb5d1) during the generation of this report, so 82dd36c was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

[Timik232]

@greptileai

[mateo-berri]

@greptileai

[mateo-berri]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 24393be. Configure here.}

[mateo-berri]

LGTM. Thanks for the contribution!