用分块线性注意力实现高效语音识别的长时流式大模型
Efficient Streaming LLM for Speech Recognition
- 分块处理音频，限制注意力窗口降低计算复杂度
- 1.28秒分块下，测试集错误率仅2.7%/6.7%，长音频无性能下降
- 适合需要实时处理长语音的场景，如会议转录、播客分析
近期研究显示，将音频编码输入大语言模型可激活语音识别能力。然而现有方法在处理长时流式音频时效率低下——不仅对训练中未见长度的音频外推能力差，且因注意力机制的二次复杂度导致计算开销大。本文提出SpeechLLM-XL，一种针对流式语音识别的线性扩展解码器模型。通过可配置分块方式处理音频，使用有限注意力窗口以减少计算量；每段音频对应文本自回归生成，直至预测到结束符。训练时，利用编码器输出估计的CTC强制对齐将转写分割为块。实验表明，采用1.28秒分块的SpeechLLM-XL在LibriSpeech测试集clean/other上的词错误率为2.7%/6.7%，且在长度为训练样本10倍的长语音上仍保持性能稳定。
原文摘要 · Abstract (English)
Recent works have shown that prompting large language models with audio encodings can unlock speech recognition capabilities. However, existing techniques do not scale efficiently, especially while handling long form streaming audio inputs -- not only do they extrapolate poorly beyond the audio length seen during training, but they are also computationally inefficient due to the quadratic cost of attention. In this work, we introduce SpeechLLM-XL, a linear scaling decoder-only model for streaming speech recognition. We process audios in configurable chunks using limited attention window for reduced computation, and the text tokens for each audio chunk are generated auto-regressively until an EOS is predicted. During training, the transcript is segmented into chunks, using a CTC forced alignment estimated from encoder output. SpeechLLM-XL with 1.28 seconds chunk size achieves 2.7%/6.7% WER on LibriSpeech test clean/other, and it shows no quality degradation on long form utterances 10x longer than the training utterances.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。