通过早期稀疏化加速语音转录，性能损失小于1%时提速1.6倍。
Early Attentive Sparsification Accelerates Neural Speech Transcription
- 在编码早期对隐藏状态进行40%-60%的稀疏化处理
- 在英语语音任务中实现最高1.6倍的运行加速，精度下降<1%
- 无需微调，适用于Whisper等Transformer模型
基于Transformer的神经语音处理已达到顶尖性能。由于语音信号具有高度可压缩性，本文通过在神经编码阶段早期进行时域信号稀疏化，以加速语音转录，并利用Transformer音频编码器中自注意力机制的可解释性。在Whisper系列模型上，我们系统地搜索了稀疏化阶段（特定编码层）与压缩率（稀疏度）的联合空间。发现当精度下降低于1%时，最优方案是在早期编码阶段将隐藏状态稀疏化至40%-60%的稀疏度，从而在Nvidia GPU上实现英语语音转录任务最高1.6倍的运行加速，且无需任何微调。
原文摘要 · Abstract (English)
Transformer-based neural speech processing has achieved state-of-the-art performance. Since speech audio signals are known to be highly compressible, here we seek to accelerate neural speech transcription by time-domain signal sparsification early in the neural encoding stage, taking advantage of the interpretability of the self-attention mechanism in transformer audio encoders. With the Whisper family of models, we perform a systematic architecture search over the joint space of sparsification stage (a certain encoder layer) and compression ratio (sparsity). We found that the best resulting solutions under 1% accuracy degradation choose to sparsify the hidden state to 40-60% sparsity at an early encoding stage, and thereby achieve up to 1.6x runtime acceleration in English speech transcription tasks on Nvidia GPUs without any fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。