用动态分段让大模型实时听中文，延迟降六成
Streaming Speech Recognition with Decoder-Only Large Language Models and Latency Optimization
- 用读写策略+单调块注意力动态切分语音
- 中文数据集上错误率5.1%~5.5%，延迟降62.5%
- 适合追求低延迟的语音识别落地应用
近期研究表明，解码器型大语言模型（LLM）在自动语音识别（ASR）中具有潜力。然而，在该框架内实现流式识别仍具挑战。本文提出一种新型流式ASR方法，结合读写策略网络与单调块注意力（MoChA），动态分割语音嵌入。这些片段在训练时与标签序列交错，实现与LLM的无缝集成。推理时，音频流被缓冲，直到MoChA模块触发读信号，此时将缓冲片段与前一标记输入LLM进行下一标记预测。我们还引入最小延迟训练目标，引导策略网络精准定位分割边界。此外，采用联合训练策略，使非流式LLM-ASR模型与本流式模型共享参数。在AISHELL-1和AISHELL-2中文基准上的实验表明，本方法持续优于近期流式ASR基线，字符错误率分别为5.1%和5.5%。延迟优化使平均标记生成延迟降低62.5%，对识别准确率影响极小。
原文摘要 · Abstract (English)
Recent advances have demonstrated the potential of decoderonly large language models (LLMs) for automatic speech recognition (ASR). However, enabling streaming recognition within this framework remains a challenge. In this work, we propose a novel streaming ASR approach that integrates a read/write policy network with monotonic chunkwise attention (MoChA) to dynamically segment speech embeddings. These segments are interleaved with label sequences during training, enabling seamless integration with the LLM. During inference, the audio stream is buffered until the MoChA module triggers a read signal, at which point the buffered segment together with the previous token is fed into the LLM for the next token prediction. We also introduce a minimal-latency training objective to guide the policy network toward accurate segmentation boundaries. Furthermore, we adopt a joint training strategy in which a non-streaming LLM-ASR model and our streaming model share parameters. Experiments on the AISHELL-1 and AISHELL-2 Mandarin benchmarks demonstrate that our method consistently outperforms recent streaming ASR baselines, achieving character error rates of 5.1% and 5.5%, respectively. The latency optimization results in a 62.5% reduction in average token generation delay with negligible impact on recognition accuracy
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。