解决语音模型训练中知识遗忘问题，提升持续学习能力。
Analyzing Mitigation Strategies for Catastrophic Forgetting in End-to-End Training of Spoken Language Models
- 用经验回放等策略缓解多阶段训练中的灾难性遗忘
- 经验回放显著提升知识保留，效果优于其他方法
- 适合需要稳定语音理解与生成能力的研究者
端到端语音语言模型（SLMs）的训练通常通过多阶段流程，将预训练文本大模型（LLMs）适配至语音模态，涵盖自动语音识别（ASR）、语音合成（TTS）和口语问答（SQA）等任务。尽管这一持续学习过程赋予大模型语音理解和生成能力，但各阶段任务与数据分布差异较大，易引发灾难性遗忘，导致先前知识丢失。本文研究了该现象，并评估了三种缓解策略：模型合并、降低LoRA缩放因子、经验回放。实验表明，经验回放效果最佳，结合其他方法可进一步提升性能。这些发现为构建更鲁棒高效的语音模型训练流程提供了重要参考。
原文摘要 · Abstract (English)
End-to-end training of Spoken Language Models (SLMs) commonly involves adapting pre-trained text-based Large Language Models (LLMs) to the speech modality through multi-stage training on diverse tasks such as ASR, TTS and spoken question answering (SQA). Although this multi-stage continual learning equips LLMs with both speech understanding and generation capabilities, the substantial differences in task and data distributions across stages can lead to catastrophic forgetting, where previously acquired knowledge is lost. This paper investigates catastrophic forgetting and evaluates three mitigation strategies-model merging, discounting the LoRA scaling factor, and experience replay to balance knowledge retention with new learning. Results show that experience replay is the most effective, with further gains achieved by combining it with other methods. These findings provide insights for developing more robust and efficient SLM training pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。