WhisperX: time-accurate speech transcription of long-form audio

Conference item

Large-scale, weakly-supervised speech recognition models, such as Whisper, have demonstrated impressive results on speech recognition across domains and languages. However, their application to long audio transcription via buffered or sliding window approaches is prone to drifting, hallucination & repetition; and prohibits batched transcription due to their sequential nature. Further, timestamps corresponding each utterance are prone to inaccuracies and word-level timestamps are not available out-of-the-box. To overcome these challenges, we present WhisperX, a time-accurate speech recognition system with word-level timestamps utilising voice activity detection and forced phoneme alignment. In doing so, we demonstrate state-of-the-art performance on long-form transcription and word segmentation benchmarks. Additionally, we show that pre-segmenting audio with our proposed VAD Cut & Merge strategy improves transcription quality and enables a twelvefold transcription speedup via batched inference. The code is available open-source.

Authors: Bain, M; Huh, J; Han, T; Zisserman, A

• Publication status: Published
• Peer review status: Peer reviewed
• Publisher: International Speech Communication Association
• Pages: 4489-4493
• Publication date: 2023-08-18
• Acceptance date: 2023-05-17
• Event title: 24th Interspeech Conference 2023
• Event location: Dublin, Ireland
• Event website: https://www.interspeech2023.org/
• Event start date: 2023-08-20
• Event end date: 2023-08-24
• DOI: 10.21437/Interspeech.2023-78
• Language: English
• Keywords: FFR