用轻量正则化让模型微调更稳定,兼顾效果与效率。
Anchored Supervised Fine-Tuning
- 在动态微调基础上加入KL约束,防止分布漂移
- 数学推理、医疗知识、代码生成全场景超越基线
- 理论分析指导实践,带来稳定提升
大语言模型后训练面临监督微调(SFT)易记忆、强化学习(RL)成本高的权衡。动态微调(DFT)虽在某些推理任务中表现优异,但存在不稳定性。本文通过奖励加权回归(RWR)框架分析发现,DFT对应特定辅助分布,可获得比SFT更紧的RL界,但缺乏分布锚定导致渐进漂移。为此提出锚定监督微调(ASFT),在DFT重加权基础上引入轻量KL正则,既保持理论紧性又确保训练稳定。实验表明,ASFT在数学推理、医疗知识对齐和代码生成任务上均显著优于SFT与DFT,且计算开销极低。RWR框架为后训练方法提供了系统理解视角,证明严谨理论分析可同时带来理论保障与实际收益。代码已开源。
原文摘要 · Abstract (English)
Post-training of large language models involves a fundamental trade-off between supervised fine-tuning (SFT), which efficiently mimics demonstrations but tends to memorize, and reinforcement learning (RL), which achieves better generalization at higher computational cost. Dynamic Fine-Tuning (DFT) recently emerged as a promising middle ground, reweighting SFT objectives with token probabilities and achieving improvements in certain reasoning domains, though it exhibits instability in other tasks. We provide a analysis of DFT through the reward-weighted regression (RWR) framework, revealing that it corresponds to a specific auxiliary distribution choice that yields provably tighter RL bounds than standard SFT. However, our analysis also uncovers a critical limitation: this construction lacks distributional anchoring, leading to progressive drift that undermines training stability. To address this, we propose Anchored Supervised Fine-Tuning (ASFT), which augments DFT's reweighting with lightweight KL regularization to preserve tightness while ensuring stability. Empirically, ASFT consistently outperforms both SFT and DFT across mathematical reasoning, medical knowledge grounding, and code generation, achieving substantial improvements with minimal computational overhead. Our RWR framework provides a systematic lens for understanding post-training methods and demonstrates that principled theoretical analysis leads to both stronger guarantees and practical gains. The code is available at https://github.com/zhuchichi56/ASFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。