Skip to content

perf(whisper): reuse encoded audio features for word timestamps - #1432

Open
unohee wants to merge 1 commit into
ml-explore:mainfrom
unohee:feat/reuse-alignment-audio-features
Open

perf(whisper): reuse encoded audio features for word timestamps#1432
unohee wants to merge 1 commit into
ml-explore:mainfrom
unohee:feat/reuse-alignment-audio-features

Conversation

@unohee

@unohee unohee commented Jul 26, 2026

Copy link
Copy Markdown

Summary

  • reuse DecodingResult.audio_features during word-timestamp alignment
  • avoid running the Whisper audio encoder a second time for every decoded window
  • preserve the existing mel-based alignment path for direct callers that do not provide encoded features

Why

transcribe() already computes and retains encoded audio features in each DecodingResult. find_alignment() discarded those features and called model.encoder(mel) again before the cross-attention/DTW pass. Passing the existing features removes that duplicate encoder invocation without changing decoding or alignment math.

M1 Max benchmark

Model: mlx-community/whisper-large-v3-turbo, FP16, Korean meeting audio, word_timestamps=True, condition_on_previous_text=False.

Audio mlx-whisper 0.4.3 This PR Speedup
65 s, warm p50 3.958 s 2.243 s 1.76x
4 min, warm 10.388 s 6.844 s 1.52x
71 min, single run 188.880 s 130.257 s 1.45x

Parity evidence

On the deterministic 4-minute run, all of the following matched exactly:

  • complete text
  • 88 segment token arrays and start/end timestamps
  • 419 word strings and start/end timestamps
  • all word probabilities

The long-form default temperature tuple can sample stochastically after a fallback. Re-running an observed fallback region with the same mx.random.seed() produced exact segment, token, timestamp, and probability parity across 19 segments and 53 words.

Scope

This PR intentionally does not include the separate batched-decoding experiment. Fixed-window batching changed long-form window boundaries and failed the output-parity gate.

@unohee
unohee marked this pull request as ready for review July 26, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant