让语音识别模型一次预测多个词，速度提升50%且准确率几乎不变
Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASR
- 每轮推理同时生成多个词，突破传统逐词生成瓶颈
- 在多种数据集上实现50%延迟降低，WER变化小于1%
- 适合实时语音转写、边缘设备部署等对速度敏感场景
基于Transformer的大规模语音识别模型具有捕捉音频序列复杂模式的潜力，但其计算开销大，导致推理速度慢。本文提出Whisper-Medusa，扩展OpenAI Whisper架构，在每轮迭代中预测多个输出词，显著提升处理速度。该方法在不同学习设置和数据集上均表现出色，实现50%的延迟降低，同时对词错误率（WER）影响极小。实验表明，该方案在保持高精度的前提下有效提升了推理效率。
原文摘要 · Abstract (English)
Large transformer-based models have significant potential for speech transcription and translation. Their self-attention mechanisms and parallel processing enable them to capture complex patterns and dependencies in audio sequences. However, this potential comes with challenges, as these large and computationally intensive models lead to slow inference speeds. Various optimization strategies have been proposed to improve performance, including efficient hardware utilization and algorithmic enhancements. In this paper, we introduce Whisper-Medusa, a novel approach designed to enhance processing speed with minimal impact on Word Error Rate (WER). The proposed model extends the OpenAI's Whisper architecture by predicting multiple tokens per iteration, resulting in a 50% reduction in latency. We showcase the effectiveness of Whisper-Medusa across different learning setups and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。