用扩散模型实现高速高精度多语言语音识别,突破传统方法速度与精度瓶颈。
Whisfusion: Parallel ASR Decoding with Masked Diffusion
- 基于冻结的Whisper音频特征,从零训练掩码扩散解码器,仅需几步去噪。
- 在11种语言上达到比Whisper-large-v3更高的平均准确率,速度快4-5倍。
- 适合追求高吞吐、多语言语音识别场景的研究者与开发者。
自回归(AR)编码器-解码器模型主导高质量多语言语音识别(ASR),但其从左到右的解码方式导致推理延迟随文本长度线性增长。非自回归(NAR)的CTC系统虽避免此瓶颈,却因条件独立性假设牺牲了文本级生成建模能力。掩码扩散语言模型(如LLaDA、MDLM)提供了一种有竞争力的NAR文本生成方法。本文探讨这类模型能否在保持高精度的同时摆脱左向瓶颈。提出Whisfusion:在冻结的Whisper-large-v3音频嵌入基础上,从头训练专用掩码扩散解码器,仅需数步即可去噪掩码文本。训练使用约68,000小时、涵盖11种语言的语音数据,并采用高掩码率以对齐推理时完全掩码的初始状态。通过并行扩散解码实现高效推理。Whisfusion在英语、欧洲语系和中日韩语基准上均超越Whisper-large-v3的组平均准确率,且速度提升4-5倍;同时优于Whisper-turbo在准确率与吞吐量上的表现。其性能媲美Canary和Qwen3-ASR,但运行速度快3-7倍。结果表明,掩码扩散是高吞吐多语言转录中帕累托最优的NAR范式。代码与模型权重已开源。
原文摘要 · Abstract (English)
Autoregressive (AR) encoder-decoder models dominate high-quality multilingual ASR, but their left-to-right decoders make inference latency scale with transcript length. A natural alternative, CTC-style non-autoregressive (NAR) systems avoid this bottleneck but their conditional independence assumption sacrifices transcript-level generative modeling. Masked diffusion language models (e.g., LLaDA, MDLM) offer a competitive NAR text-generation approach. We ask whether such models can bring NAR ASR into the accuracy regime of strong AR ASR systems while removing the left-to-right bottleneck. We propose Whisfusion, which trains a dedicated masked diffusion decoder from scratch on top of frozen Whisper-large-v3 audio embeddings, denoising masked transcripts in just a few steps. We train on ~68k hours of 11-language speech with high-mask specialization to align training with the fully masked starting point of inference, and decode via Parallel Diffusion Decoding. Whisfusion surpasses Whisper-large-v3 on group-average accuracy across English, European, and CJK benchmarks, while running 4-5x faster, additionally surpassing Whisper-turbo in both accuracy and throughput. It reaches accuracy competitive with Canary and Qwen3-ASR while running 3-7x faster. These results establish masked diffusion as a Pareto-competitive non-autoregressive paradigm for high-throughput multilingual transcription. Code and model weights are available at https://github.com/taeyoun811/Whisfusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。