通过动态锚点控制分布变化,解决大模型微调中的灾难性遗忘问题。
Stabilizing LLM Supervised Fine-Tuning via Explicit Distributional Control

- 用动态锚点构建中间目标,将全局微调转为局部信任区域更新。
- 在iGSM、MedCalc等数据集上性能下降低于5%,比标准微调降低超48%。
- 适合追求稳定性能提升的工业级LLM微调场景。
后训练大语言模型常面临灾难性遗忘,即目标任务优化导致已有能力显著退化。近期研究表明,这主要由优化过程中的过度分布漂移引起。为此,我们提出锚定学习(Anchored Learning),一种通过动态演化的移动锚点显式控制分布更新的简单框架。该方法不匹配固定参考分布,而是将当前模型与冻结参考分布进行插值,构造一个中间目标,使模型向其蒸馏,从而将全局微调转化为分布空间中一系列局部信任区域更新。理论上,我们证明该锚点更新每步具有线性KL散度上界,确保模型分布间稳定过渡。在iGSM、MedCalc和IFEval上的大量实验表明,锚定学习始终位于增益-稳定性权衡的帕累托前沿:在保持近似最优性能提升的同时,显著降低退化。例如,标准SFT在iGSM和MedCalc上性能下降超过53%,而锚定学习将其降至不足5%,同时保持高精度(如iGSM达75.2%)。
原文摘要 · Abstract (English)
Post-training large language models (LLMs) often suffers from catastrophic forgetting, where improvements on a target objective degrade previously acquired capabilities. Recent evidence suggests that this phenomenon is primarily driven by excessive distributional drift during optimization. Motivated by this perspective, we propose Anchored Learning, a simple framework that explicitly controls distributional updates during offline fine-tuning via a dynamically evolving moving anchor. Instead of matching a fixed reference distribution, the anchor interpolates between the current model and a frozen reference to construct an intermediate target that the model distills toward, transforming global fine-tuning into a sequence of local trust-region updates in distribution space. Theoretically, we prove this anchor-based update admits a linear KL-divergence upper bound per iteration, ensuring a stable transition between model distributions. Extensive experiments on iGSM, MedCalc, and IFEval show that Anchored Learning consistently lies on the Pareto frontier of gain-stability trade-offs, achieving near-optimal performance improvements while substantially reducing degradation compared to strong baselines. For example, while standard SFT suffers from over 53% performance degradation on iGSM and MedCalc, Anchored Learning slashes this drop to under 5% while maintaining near-optimal gains (e.g., 75.2% on iGSM).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。