让Whisper支持流式语音识别，通过双阶段解码提升实时性能。
Adapting Whisper for Streaming Speech Recognition via Two-Pass Decoding
- 采用双阶段解码结构，用CTC生成流式中间结果。
- 在LibriSpeech和财报电话数据集上实现高效流式识别。
- 混合分词策略提升数据效率，适合资源受限场景。
OpenAI Whisper 是一系列基于68万小时音频训练的鲁棒自动语音识别（ASR）模型。然而，其编码器-解码器架构在序列到序列目标函数下缺乏对流式ASR的原生支持。本文通过WeNet工具链，采用统一双阶段（U2）结构对Whisper进行微调，以实现流式ASR。我们引入一个使用因果注意力掩码训练的连接时序分类（CTC）解码器，生成流式部分转录文本，而原始Whisper解码器则对这些部分输出进行重排序。在LibriSpeech和财报电话数据集上的实验表明，经过充分微调后，Whisper可转化为具备竞争力的流式ASR系统。此外，我们提出一种混合分词方法：为CTC解码器使用更小的词表空间，同时保留Whisper原有的词表空间用于注意力解码器，从而提升数据效率与泛化能力。
原文摘要 · Abstract (English)
OpenAI Whisper is a family of robust Automatic Speech Recognition (ASR) models trained on 680,000 hours of audio. However, its encoder-decoder architecture, trained with a sequence-to-sequence objective, lacks native support for streaming ASR. In this paper, we fine-tune Whisper for streaming ASR using the WeNet toolkit by adopting a Unified Two-pass (U2) structure. We introduce an additional Connectionist Temporal Classification (CTC) decoder trained with causal attention masks to generate streaming partial transcripts, while the original Whisper decoder reranks these partial outputs. Our experiments on LibriSpeech and an earnings call dataset demonstrate that, with adequate fine-tuning data, Whisper can be adapted into a capable streaming ASR model. We also introduce a hybrid tokenizer approach, which uses a smaller token space for the CTC decoder while retaining Whisper's original token space for the attention decoder, resulting in improved data efficiency and generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。