arXiv:2605.04468cs.LGcs.AI2026-05

通过动态锚点控制分布变化,解决大模型微调中的灾难性遗忘问题。

Stabilizing LLM Supervised Fine-Tuning via Explicit Distributional Control

论文配图:Stabilizing LLM Supervised Fine-Tuning via Explicit Distributional Control
图 1 · 摘自论文原文
  • 用动态锚点构建中间目标,将全局微调转为局部信任区域更新。
  • 在iGSM、MedCalc等数据集上性能下降低于5%,比标准微调降低超48%。
  • 适合追求稳定性能提升的工业级LLM微调场景。

后训练大语言模型常面临灾难性遗忘,即目标任务优化导致已有能力显著退化。近期研究表明,这主要由优化过程中的过度分布漂移引起。为此,我们提出锚定学习(Anchored Learning),一种通过动态演化的移动锚点显式控制分布更新的简单框架。该方法不匹配固定参考分布,而是将当前模型与冻结参考分布进行插值,构造一个中间目标,使模型向其蒸馏,从而将全局微调转化为分布空间中一系列局部信任区域更新。理论上,我们证明该锚点更新每步具有线性KL散度上界,确保模型分布间稳定过渡。在iGSM、MedCalc和IFEval上的大量实验表明,锚定学习始终位于增益-稳定性权衡的帕累托前沿:在保持近似最优性能提升的同时,显著降低退化。例如,标准SFT在iGSM和MedCalc上性能下降超过53%,而锚定学习将其降至不足5%,同时保持高精度(如iGSM达75.2%)。

原文摘要 · Abstract (English)

Post-training large language models (LLMs) often suffers from catastrophic forgetting, where improvements on a target objective degrade previously acquired capabilities. Recent evidence suggests that this phenomenon is primarily driven by excessive distributional drift during optimization. Motivated by this perspective, we propose Anchored Learning, a simple framework that explicitly controls distributional updates during offline fine-tuning via a dynamically evolving moving anchor. Instead of matching a fixed reference distribution, the anchor interpolates between the current model and a frozen reference to construct an intermediate target that the model distills toward, transforming global fine-tuning into a sequence of local trust-region updates in distribution space. Theoretically, we prove this anchor-based update admits a linear KL-divergence upper bound per iteration, ensuring a stable transition between model distributions. Extensive experiments on iGSM, MedCalc, and IFEval show that Anchored Learning consistently lies on the Pareto frontier of gain-stability trade-offs, achieving near-optimal performance improvements while substantially reducing degradation compared to strong baselines. For example, while standard SFT suffers from over 53% performance degradation on iGSM and MedCalc, Anchored Learning slashes this drop to under 5% while maintaining near-optimal gains (e.g., 75.2% on iGSM).

大模型微调分布控制稳定性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。