arXiv:2607.29279cs.SD2026-07被引 1

用多标记预测加速语音识别,30分钟音频单次完成,延迟极低。

ParaASR: Multi-Token Prediction for Fast and Long-Context LLM-Based Speech Recognition

论文配图:ParaASR: Multi-Token Prediction for Fast and Long-Context LLM-Based Speech Recognition
图 1 · 摘自论文原文
  • 通过多标记预测让40亿参数大模型每步输出多个词,大幅提升速度
  • 在中文、英文和长语音上误差率分别仅2.97%、3.68%、3.70%,实时因子低至0.0053
  • 基于声学信号锚定未来生成,确保准确性,适合长时语音高并发场景

音频编码器-大语言模型-解码器架构已成为现代语音识别主流,借助大规模语言建模提升转录质量。但自回归解码的开销随解码器规模增长,造成识别质量与服务延迟的根本权衡。我们提出:该权衡并非必然——与开放式文本生成不同,语音识别输出受输入语音强约束,天然适合高并行解码。基于此,我们提出ParaASR,利用多标记预测(MTP)使40亿参数的LLM解码器每前向传播一步输出多个标记。从公开可用的音-语基础模型出发,先建立稳健的自回归识别器,再通过分阶段优化策略对五个未来标记分支进行对齐。推理时每步提出六标记延续,仅将经验证的前缀写入转录结果,保持标准自回归解码的安全性。平均接受长度达5.0/6,证明语音的确定性结构使多标记解码尤为适配。ParaASR保留原生32K上下文窗口,可单次处理长达30分钟音频。在多个基准测试中,中文、英文及长语音评估的平均错误率分别为2.97%、3.68%、3.70%,实时因子(RTF)低至0.0053。结果表明,当未来标记预测由声学信号锚定并由自回归验证防护时,解码器扩展、低延迟推理与长上下文转录可兼得。

原文摘要 · Abstract (English)

Audio-encoder-LLM-decoder architectures have become the dominant paradigm for modern automatic speech recognition (ASR), improving transcription quality through large-scale language modeling. However, the cost of autoregressive decoding scales with decoder size, creating a fundamental trade-off between recognition quality and serving latency. We argue this trade-off is not inherent: unlike open-ended text generation, ASR outputs are strongly anchored to the input speech signal, providing a natural inductive bias toward high-parallelism decoding. Building on this, we introduce ParaASR, an ASR system that leverages Multi-Token Prediction (MTP) to let a 4B LLM decoder emit multiple tokens per forward step. Starting from a publicly available audio-language foundation, the model first establishes a robust autoregressive recognizer and then aligns five future-token branches through a staged optimization recipe. At inference, it proposes a six-token continuation per step and admits only the verified prefix into the transcript, preserving the safety of standard autoregressive decoding. The average accepted length reaches 5.0 out of 6 proposed tokens, confirming that the deterministic structure of speech makes ASR an especially natural setting for multi-token decoding. ParaASR further retains a native 32K-context window and transcribes up to 30 minutes of audio in a single pass. Across diverse benchmarks, it attains average error rates of 2.97%, 3.68%, and 3.70% on Chinese, English, and long-form evaluations, respectively, while reaching a real-time factor (RTF) as low as 0.0053. These results show that decoder scaling, low-latency inference, and long-context transcription need not be competing goals when future-token proposals are anchored by the acoustic signal and guarded by autoregressive verification.

语音识别多标记预测长上下文低延迟

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。