通过双向上下文增强，提升多语言对话语音识别准确率
Bi-directional Context-Enhanced Speech Large Language Models for Multilingual Conversational ASR
- 用字符级掩码训练，模拟实际转录错误
- 两阶段解码使识别率相对基线提升18%
- 适合需要高精度多语种语音识别的场景
本文提出在语音大模型中引入语言特异性双向上下文，以提升多语言连续对话自动语音识别（ASR）性能。训练时采用字符级上下文掩码策略，随机移除部分上下文内容，增强模型鲁棒性并更真实模拟推理时可能出现的转录错误。解码采用两阶段流程：先对孤立片段进行初步解码，再利用邻近假设进行上下文感知重解码。在包含11种语言的1500小时多语言对话语音与语言模型（MLC-SLM）数据集上评估，该方法相比强基线实现18%的相对性能提升，甚至超过在6000小时数据上训练的模型在MLC-SLM竞赛中的表现。结果表明，在多语言连续对话ASR中融入上下文信息具有显著优势。
原文摘要 · Abstract (English)
This paper introduces the integration of language-specific bi-directional context into a speech large language model (SLLM) to improve multilingual continuous conversational automatic speech recognition (ASR). We propose a character-level contextual masking strategy during training, which randomly removes portions of the context to enhance robustness and better emulate the flawed transcriptions that may occur during inference. For decoding, a two-stage pipeline is utilized: initial isolated segment decoding followed by context-aware re-decoding using neighboring hypotheses. Evaluated on the 1500-hour Multilingual Conversational Speech and Language Model (MLC-SLM) corpus covering eleven languages, our method achieves an 18% relative improvement compared to a strong baseline, outperforming even the model trained on 6000 hours of data for the MLC-SLM competition. These results underscore the significant benefit of incorporating contextual information in multilingual continuous conversational ASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。