Repository navigation
[Perf] Fix /audio/speech performance by using shared_sessions - #16739
Merged
Merged
Conversation
1KB is too small for audio and is lowering the RPS when testing with medium to large files
Refactor async_data_generator to build streamed text via list accumulation and ''.join() instead of repeated string concatenation. This improves performance for long responses without changing streaming behavior.
Use list accumulation and join in get_response_string to avoid O(n^2) string concatenation and add a brief comment explaining the performance rationale.
The optimizations related to `select_data_generator` had no effect because its output which is the generator wasn't being used.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
For anybody that would change this value for whatever reason, the comment makes the tradeoff clear.
ghost
reviewed
Nov 18, 2025
There was a problem hiding this comment.
can we avoid having test files on the root of the repo? @AlexsanderHamir
Contributor
Author
There was a problem hiding this comment.
Thank you @krrishdholakia, I forgot to remove it.
|
minor cleanup on test files needed |
ishaan-jaff
suggested changes
Nov 18, 2025
| # too small: latency is high | ||
| # too large: latency is low, but memory usage is high | ||
| # 8192 is a good compromise | ||
| _generator = await _response.aiter_bytes(chunk_size=8192) |
Contributor
There was a problem hiding this comment.
8192 should be in constants
Contributor
Author
There was a problem hiding this comment.
Thank you, done!
fzowl
pushed a commit
to fzowl/litellm
that referenced
this pull request
Jun 24, 2026
[Perf] Fix `/audio/speech` performance by using `shared_sessions`
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title
[Perf] Fix
/audio/speechperformance by usingshared_sessionsRelevant issues
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unitType
🧹 Refactoring
Changes
select_data_generatorwas dead code adding overheadTest Results
Performance Gains
/audio/speechendpoint our second fastest endpoint after/realtime.