让大模型更适配流式语音识别，降低延迟同时保持高精度。
MFLA: Monotonic Finite Look-ahead Attention for Streaming Speech Recognition
- 用连续积分-放电机制实现语音与文字的单调对齐。
- 引入有限前瞻注意力，左无限、右有限地关注上下文。
- 适合对延迟敏感的实时语音识别场景，如会议转录。
将 Whisper 等大规模预训练语音模型应用于各类语音任务已展现出降低训练成本的潜力。然而，将其集成到流式系统中仍面临挑战。本文提出一种新的前缀到前缀微调框架，通过精细调整 Whisper 实现流式识别。我们引入连续积分-放电机制，建立连续语音序列与离散文本标记间的准单调对齐关系。同时设计单调有限前瞻注意力机制，使每个标记可接收无限左文和有限右文的语音信息。此外，采用 wait-k 解码策略，在简化解码过程的同时保证训练与测试的一致性。理论分析与实验表明，该方法可在延迟与识别质量间实现可控权衡，适用于多种流式应用场景。
原文摘要 · Abstract (English)
Applying large pre-trained speech models like Whisper has shown promise in reducing training costs for various speech tasks. However, integrating these models into streaming systems remains a challenge. This paper presents a novel prefix-to-prefix training framework for streaming recognition by fine-tuning the Whisper. We introduce the Continuous Integrate-and-Fire mechanism to establish a quasi-monotonic alignment between continuous speech sequences and discrete text tokens. Additionally, we design Monotonic Finite Look-ahead Attention, allowing each token to attend to infinite left-context and finite right-context from the speech sequences. We also employ the wait-k decoding strategy to simplify the decoding process while ensuring consistency between training and testing. Our theoretical analysis and experiments demonstrate that this approach achieves a controllable trade-off between latency and quality, making it suitable for various streaming applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。