用交叉注意力提升多语言语音识别中的语言特征融合效果
CAMEL: Cross-Attention Enhanced Mixture-of-Experts and Language Bias for Code-Switching Speech Recognition
- 在专家混合模型后使用交叉注意力融合语言特征
- 在多个数据集上达到当前最优性能
- 适合研究多语言语音识别与注意力机制的学者
代码切换自动语音识别（ASR）旨在准确转录包含两种或以上语言的语音。为更好捕捉语言特异性语音表征并缓解代码切换中的语言混淆问题，通常采用专家混合（MoE）架构和额外的语言辨识（LD）解码器。然而，多数研究仍停留在加权求和或拼接等简单操作上，未能充分挖掘语言偏置信息的融合潜力。本文提出CAMEL，一种基于交叉注意力的MoE与语言偏置方法，用于代码切换语音识别。具体而言，在每个MoE层后，通过交叉注意力融合语言特异性语音表征，利用其强大的上下文建模能力；此外，设计基于源注意力的机制，将LD解码器输出的语言信息融入文本嵌入中。实验结果表明，该方法在SEAME、ASRU200和ASRU700+LibriSpeech460中文-英文代码切换语音识别数据集上均达到当前最优性能。
原文摘要 · Abstract (English)
Code-switching automatic speech recognition (ASR) aims to transcribe speech that contains two or more languages accurately. To better capture language-specific speech representations and address language confusion in code-switching ASR, the mixture-of-experts (MoE) architecture and an additional language diarization (LD) decoder are commonly employed. However, most researches remain stagnant in simple operations like weighted summation or concatenation to fuse languagespecific speech representations, leaving significant opportunities to explore the enhancement of integrating language bias information. In this paper, we introduce CAMEL, a cross-attention-based MoE and language bias approach for code-switching ASR. Specifically, after each MoE layer, we fuse language-specific speech representations with cross-attention, leveraging its strong contextual modeling abilities. Additionally, we design a source attention-based mechanism to incorporate the language information from the LD decoder output into text embeddings. Experimental results demonstrate that our approach achieves state-of-the-art performance on the SEAME, ASRU200, and ASRU700+LibriSpeech460 Mandarin-English code-switching ASR datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。