arXiv:2508.18998eess.AS2025-08中稿 · ICASSP 2026被引 5

用多个简单适配器混合提升多语言语音识别效果,更省参数且抗数据不均衡。

MOSA: Mixtures of Simple Adapters Outperform Monolithic Approaches in LLM-based Multilingual ASR

  • 设计多专家适配器混合架构,让不同专家专攻共性或特定语言特征。
  • 参数减少40%仍比单适配器低13.3%错误率,平均错词率降15.4%。
  • 适合资源少的语言,特别适合追求高效与鲁棒性的多语言语音系统。

基于大语言模型的语音识别通过将语音表示映射到语言模型空间,利用其强大的语义和推理能力缓解多语言数据稀缺问题。然而,以往方法通常依赖扩大数据或模型参数,而单一投影器难以有效对齐不同语言间的表示。本文提出一种基于门控专家混合(MoE)的投影器MOSA(Mixture of Simple Adapters)。该架构通过聚合多个简单适配器,使不同专家分别学习语言共享或语言特异性知识,从而减轻语言间参数干扰,并促进高资源语言向低资源语言的正向迁移,有效缓解数据稀缺问题。实验表明,MOSA-Base相比Ideal-LLM Base在平均字错误率(WER)上降低15.4%,且仅使用其60%参数即实现13.3%的WER下降。结果表明,MOSA在参数效率和应对数据不平衡方面表现更优,证明简单适配器的混合结构比复杂单适配器更适合多语言大模型语音识别。

原文摘要 · Abstract (English)

LLM-based ASR overcomes multilingual data scarcity by projecting speech representations into the LLM space to leverage its robust semantic and reasoning capabilities. However, while previous approaches typically enhance performance by scaling data or model parameters, a single projector often struggles to effectively align representations across different languages. In this work, we propose an MoE-based projector named MOSA (Mixture of Simple Adapters). By aggregating multiple simple adapters, this architecture enables different experts to specialize in learning either language-shared or language-specific knowledge. This approach not only mitigates parameter interference between languages but also facilitates positive transfer from high-resource to low-resource languages, effectively alleviating data scarcity issues. Experimental results demonstrate that MOSA-Base achieves a 15.4% relative reduction in average WER compared to the Ideal-LLM Base, consistently outperforming it across all languages. Notably, MOSA achieves a 13.3% WER reduction over the Ideal-LLM Base while utilizing only 60% of its parameters. These findings highlight MOSA's superior parameter efficiency and robustness against data imbalance, suggesting that a mixture of simple adapters is more suitable for multilingual LLM-based ASR than complex single-adapter designs.

语音识别多语言适配器高效模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。