融合双编码器与大模型，实现多语种对话语音识别新突破
SHNU Multilingual Conversational Speech Recognition System for INTERSPEECH 2025 MLC-SLM Challenge
- 双预训练编码器并行处理，输出拼接后输入大模型
- 三阶段联合微调，提升多语言识别准确率
- 加语言提示增强特定语言生成，适合多语种场景应用
本文介绍上海师范大学提交至INTERSPEECH 2025 MLC-SLM挑战赛第1赛道的多语种对话语音识别系统（SHNU-mASR，团队名：maybe）。该系统采用并行语音编码器架构结合大语言模型（LLM），构建统一的多语种自动语音识别框架。并行编码器由Whisper-large-v3和mHuBERT-147两个预训练模型组成，其输出嵌入拼接后输入LLM，使模型可融合互补的声学与语言知识，实现优异性能。此外，采用三阶段训练策略，联合更新语音编码器与LLM中的低秩适配模块及投影参数。同时，在LLM输入中加入语言感知提示，增强语言特异性文本生成能力。该系统在挑战赛盲评集上取得11.76%的总体字符错误率（CER）/词错误率（WER），相较官方基准绝对降低8.41个百分点，且未增加基线训练数据。
原文摘要 · Abstract (English)
This paper describes SHNU multilingual conversational speech recognition system (SHNU-mASR, team name-"maybe"), submitted to Track 1 of the INTERSPEECH 2025 MLC-SLM Challenge. Our system integrates a parallel-speech-encoder architecture with a large language model (LLM) to form a unified multilingual ASR framework. The parallel-speech-encoder consists of two pre-trained encoders, the Whisper-large-v3 encoder and mHuBERT-147 encoder. Their output embeddings are concatenated and fed into the LLM, enabling the model to leverage complementary acoustic and linguistic knowledge and achieve competitive performance. Moreover, we adopt a tri-stage training strategy to jointly update the low-rank adaptation modules and projector parameters of both the speech encoders and the LLM. In addition, we incorporate an additional language-aware prompt at the LLM input to enhance language-specific text generation. The SHNU-mASR system achieves an overall character/word error rate (CER/WER) of 11.76% on the blind evaluation set of the challenge, outperforming the official MLC-SLM baseline by 8.41 absolute CER/WER, without increasing the baseline training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。