用多个特定口音的LoRA专家混合，提升低资源多口音语音识别鲁棒性。
Mixture of LoRA Experts for Low-Resourced Multi-Accent Automatic Speech Recognition
- 通过口音专属LoRA专家混合实现无需重新微调的适应。
- 口音未知时，词错误率降低23.7%，已知时进一步提升。
- 减少灾难性遗忘，适合低资源非母语语音识别场景。
我们旨在提升自动语音识别（ASR）系统对非母语语音的鲁棒性，尤其在低资源多口音场景下。提出一种基于低秩适配（LoRA）专家混合的音调特异性方法（MAS-LoRA），每个专家针对特定口音进行优化。该方法在推理时无论口音是否已知均适用，无需再次微调模型。在Whisper和L2-ARCTIC数据集上的实验表明，当口音未知时，相比常规LoRA和全量微调，词错误率显著下降；当口音已知时性能进一步提升。此外，与其它微调方法相比，MAS-LoRA表现出更少的灾难性遗忘。据我们所知，这是首次将LoRA专家混合用于非母语多口音语音识别。
原文摘要 · Abstract (English)
We aim to improve the robustness of Automatic Speech Recognition (ASR) systems against non-native speech, particularly in low-resourced multi-accent settings. We introduce Mixture of Accent-Specific LoRAs (MAS-LoRA), a fine-tuning method that leverages a mixture of Low-Rank Adaptation (LoRA) experts, each specialized in a specific accent. This method can be used when the accent is known or unknown at inference time, without the need to fine-tune the model again. Our experiments, conducted using Whisper on the L2-ARCTIC corpus, demonstrate significant improvements in Word Error Rate compared to regular LoRA and full fine-tuning when the accent is unknown. When the accent is known, the results further improve. Furthermore, MAS-LoRA shows less catastrophic forgetting than the other fine-tuning methods. To the best of our knowledge, this is the first use of a mixture of LoRA experts for non-native multi-accent ASR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。