通过分语言嵌入与任务导向搜索，缓解语音模型学新语言时遗忘旧语言的问题。
Continual Learning with Embedding Layer Surgery and Task-wise Beam Search using Whisper
- 为每种语言创建独立词元嵌入副本，按需替换以减少遗忘
- 在10种新语言上训练后，旧语言平均错误率从14.2%降至11.9%
- 适合需持续扩展语言支持的多语种语音识别系统
当前多语言语音识别模型仅支持全球少数语言。持续学习（CL）旨在向预训练模型添加新语言，同时避免对已有语言性能的损失，即灾难性遗忘（CF）。然而，现有方法忽略了解码器中词元嵌入查找表的适应性，而该部分对CF有显著影响。本文提出嵌入层手术（Embedding Layer Surgery），为每种新语言创建独立的词元嵌入副本，推理时根据语言选择对应副本替换旧嵌入。但此法可能导致语言识别错误引发嵌入选错。为此提出任务导向束搜索（Task-wise Beam Search），实现自我纠正。在Common Voice中对每种未见语言使用10小时数据微调Whisper，结果表明，相比经验回放，本方法将预训练语言的平均字错率（AWER）从14.2%降至11.9%，且未牺牲未见语言的平均性能。
原文摘要 · Abstract (English)
Current Multilingual ASR models only support a fraction of the world's languages. Continual Learning (CL) aims to tackle this problem by adding new languages to pre-trained models while avoiding the loss of performance on existing languages, also known as Catastrophic Forgetting (CF). However, existing CL methods overlook the adaptation of the token embedding lookup table at the decoder, despite its significant contribution to CF. We propose Embedding Layer Surgery where separate copies of the token embeddings are created for each new languages, and one of the copies is selected to replace the old languages embeddings when transcribing the corresponding new language. Unfortunately, this approach means LID errors also cause incorrect ASR embedding selection. Our Task-wise Beam Search allows self-correction for such mistakes. By adapting Whisper to 10 hours of data for each of 10 unseen languages from Common Voice, results show that our method reduces the Average WER (AWER) of pre-trained languages from 14.2% to 11.9% compared with Experience Replay, without compromising the AWER of the unseen languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。