用贝叶斯低秩分解减少语音模型微调时的过拟合和遗忘。
Bayesian Low-Rank Factorization for Robust Model Adaptation
- 在适配器中引入零附近先验，实现稀疏化参数更新。
- 相比LoRA，新域性能仅降4%，回溯性能提升54%。
- 适合需要保留通用能力又需适应本地语音场景的研究者。
大型语音基础模型在多个领域表现优异，但常需适配以应对如语码转换等本地需求。直接微调易导致过拟合并破坏模型泛化能力。本文提出贝叶斯因子化适配器，在适配矩阵中设置近零先验，实现更稀疏的参数更新，从而在适应特定领域的同时保持模型通用性。我们将其应用于Whisper模型，并在多种多语言语码转换场景下评估。结果表明，该方法仅造成轻微适应损失，显著缓解灾难性遗忘。相较于LoRA，本方法在新域性能仅下降4%的情况下，回溯性能提升54%。这验证了贝叶斯适配在不牺牲泛化能力的前提下有效微调语音基础模型的可行性。
原文摘要 · Abstract (English)
Large speech foundation models achieve strong performance across many domains, but they often require adaptation to handle local needs such as code-switching, where speakers mix languages within the same utterance. Direct fine-tuning of these models risks overfitting to the target domain and overwriting the broad capabilities of the base model. To address this challenge, we explore Bayesian factorized adapters for speech foundation models, which place priors near zero to achieve sparser adaptation matrices and thereby retain general performance while adapting to specific domains. We apply our approach to the Whisper model and evaluate on different multilingual code-switching scenarios. Our results show only minimal adaptation loss while significantly reducing catastrophic forgetting of the base model. Compared to LoRA, our method achieves a backward gain of 54% with only a 4% drop on the new domain. These findings highlight the effectiveness of Bayesian adaptation for fine-tuning speech foundation models without sacrificing generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。