微调大模型时，加个简单正则化就能防能力崩溃。
The Effectiveness of Approximate Regularized Replay for Efficient Supervised Fine-Tuning of Large Language Models
- 用近似回放加KL正则化，防止微调时遗忘旧知识。
- 在小数据上训练也能保持模型通用能力，不退化。
- 适合想高效微调又怕灾难性遗忘的研究者和工程师。
尽管参数高效微调方法（如LoRA）仅修改少量参数，仍可能对模型造成显著影响。我们的指令微调实验表明，即使在极小数据集上进行少量训练步骤，基于LoRA的监督微调也可能导致模型能力灾难性下降。然而，我们发现最直接的方法虽失败严重，但通过训练过程的微小调整（几乎无额外开销）即可基本消除该问题。本文提出一种正则化的近似回放方法：以初始模型为基准，惩罚与之的KL散度，并交错加入来自不同但相似开源语料库的下一项预测数据，用于预训练之外的补充。应用于Qwen指令微调模型时，该方法在引入适度计算开销的前提下，有效保留了模型的通用知识，同时不影响其对新任务的学习能力。
原文摘要 · Abstract (English)
Although parameter-efficient fine-tuning methods, such as LoRA, only modify a small subset of parameters, they can have a significant impact on the model. Our instruction-tuning experiments show that LoRA-based supervised fine-tuning can catastrophically degrade model capabilities, even when trained on very small datasets for relatively few steps. With that said, we demonstrate that while the most straightforward approach (that is likely the most used in practice) fails spectacularly, small tweaks to the training procedure with very little overhead can virtually eliminate the problem. Particularly, in this paper we consider a regularized approximate replay approach which penalizes KL divergence with respect to the initial model and interleaves in data for next token prediction from a different, yet similar, open access corpus to what was used in pre-training. When applied to Qwen instruction-tuned models, we find that this recipe preserves general knowledge in the model without hindering plasticity to new tasks by adding a modest amount of computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。