用大模型统一解决多人重叠语音和生僻词识别问题
CMT-LLM: Contextual Multi-Talker ASR Utilizing Large Language Models
- 将多说话人语音识别与上下文词偏置融合为单一任务
- 在LibriMix上实现7.9%的错误率，AMI SDM达32.9%
- 适合需要高精度复杂场景语音识别的应用
在真实应用中，自动语音识别系统需处理多说话人重叠语音并识别技术术语等罕见词汇。传统方法分别处理多说话人识别与上下文偏置，限制了复杂场景下的性能。本文提出统一框架，将多说话人重叠语音识别与上下文偏置整合为单一任务。方法结合预训练语音编码器与大语言模型（LLMs），采用优化微调策略，并设计两阶段过滤算法，高效从大规模偏置列表中筛选相关罕见词，融入LLM提示输入，提升罕见词识别能力。实验表明，该方法优于传统上下文偏置方法，在偏置列表大小为1,000时，于LibriMix上达到7.9%的单词错误率（WER），在AMI SDM上达到32.9%，证明其在复杂语音场景中的有效性。
原文摘要 · Abstract (English)
In real-world applications, automatic speech recognition (ASR) systems must handle overlapping speech from multiple speakers and recognize rare words like technical terms. Traditional methods address multi-talker ASR and contextual biasing separately, limiting performance in complex scenarios. We propose a unified framework that combines multi-talker overlapping speech recognition and contextual biasing into a single task. Our ASR method integrates pretrained speech encoders and large language models (LLMs), using optimized finetuning strategies. We also introduce a two-stage filtering algorithm to efficiently identify relevant rare words from large biasing lists and incorporate them into the LLM's prompt input, enhancing rare word recognition. Experiments show that our approach outperforms traditional contextual biasing methods, achieving a WER of 7.9% on LibriMix and 32.9% on AMI SDM when the biasing size is 1,000, demonstrating its effectiveness in complex speech scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。