arXiv:2601.01461cs.CLcs.SD2026-01中稿 · ICASSP2026

对比LLM与端到端模型在多语言语音识别中的表现,发现后者仍更优。

Bridging the gap: A comparative exploration of Speech-LLM and end-to-end architecture for multilingual conversational ASR

  • 用交叉注意力融合Whisper和mHuBERT特征,提升语音表示
  • 仅用1500小时数据达到10.69%的字符错误率,媲美顶尖系统
  • 实证表明当前Speech-LLM仍不及微调的端到端模型

INTERSPEECH 2025多语言对话语音语言模型挑战赛推动了基于大语言模型(LLM)的多语言对话语音识别发展。我们此前的SHNU-mASR系统采用并行语音编码器架构,结合Whisper与mHuBERT及一个LLM。但该方法存在两个问题:简单特征拼接难以充分挖掘互补信息,且基于LLM的语音识别与端到端(E2E)编码器-解码器模型之间的性能差距尚未明确。本文提出增强型LLM语音识别框架,通过微调Whisper和mHuBERT编码器并结合LLM以丰富语音表征。首先在MLC-SLM语音识别任务上评估了使用LoRA与全量微调的E2E Whisper模型;随后提出基于交叉注意力的并行编码器融合机制。在官方评测集上,本系统取得10.69%的字符错误率(CER)/词错误率(WER),与排名第一的Track 1系统水平相当,尽管仅使用1,500小时基础训练数据,远少于其他参赛系统的海量数据。然而,我们发现最终的LLM语音识别系统仍未达到微调后E2E Whisper模型的性能,为未来Speech-LLM设计提供了重要实证参考。代码已开源:https://github.com/1535176727/MLC-SLM。

原文摘要 · Abstract (English)

The INTERSPEECH 2025 Challenge on Multilingual Conversational Speech Language Models (MLC-SLM) promotes multilingual conversational ASR with large language models (LLMs). Our previous SHNU-mASR system adopted a competitive parallel-speech-encoder architecture that integrated Whisper and mHuBERT with an LLM. However, it faced two challenges: simple feature concatenation may not fully exploit complementary information, and the performance gap between LLM-based ASR and end-to-end(E2E) encoder-decoder ASR remained unexplored. In this work, we present an enhanced LLM-based ASR framework that combines fine-tuned Whisper and mHuBERT encoders with an LLM to enrich speech representations. We first evaluate E2E Whisper models with LoRA and full fine-tuning on the MLC-SLM ASR task, and then propose cross-attention-based fusion mechanisms for the parallel-speech-encoder. On the official evaluation set of the MLC-SLM Challenge, our system achieves a CER/WER of 10.69%, ranking on par with the top-ranked Track 1 systems, even though it uses only 1,500 hours of baseline training data compared with their large-scale training sets. Nonetheless, we find that our final LLM-based ASR still does not match the performance of a fine-tuned E2E Whisper model, providing valuable empirical guidance for future Speech-LLM design. Our code is publicly available at https://github.com/1535176727/MLC-SLM.

语音识别多语言LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。