改进微调方法，让大模型推理能力更强
On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
- 用动态重缩放优化梯度更新，修复标准微调缺陷
- 在数学、编程等任务上超越传统微调，泛化能力提升
- 代码简洁易用，适合想提升模型推理能力的研究者
本文从强化学习视角分析了大语言模型监督微调（SFT）泛化能力有限的根源。数学分析表明，标准SFT的梯度隐含了不利的奖励结构，限制了模型泛化性能。为此，我们提出动态微调（DFT），通过根据每个词元的概率动态调整目标函数，稳定梯度更新。仅需一行代码修改，DFT在多个困难基准测试中优于标准SFT，涵盖数学推理、代码生成及多模态任务，并在离线强化学习设置中取得有竞争力的结果，提供了一种高效且简洁的替代方案。该工作将理论洞察与实际应用结合，推动了SFT的发展。源代码将在https://github.com/yongliang-wu/DFT公开。
原文摘要 · Abstract (English)
In this work, we present a simple yet theoretically motivated improvement to Supervised Fine-Tuning (SFT) for the Large Language Model (LLM), addressing its limited generalization compared to reinforcement learning (RL). Through mathematical analysis, we reveal that standard SFT gradients implicitly encode a problematic reward structure that may severely restrict the generalization capabilities of model compared to RL. To rectify this, we propose Dynamic Fine-Tuning (\model), stabilizing gradient updates for each token by dynamically rescaling the objective function with the probability of this token. With just a single-line change, the method outperforms standard SFT on multiple difficult benchmarks and base models, from math reasoning to code generation and multi-modal tasks, demonstrating improved generalization. Additionally, \model~achieves competitive results in offline RL settings, providing an effective yet streamlined alternative. By bridging theoretical insights with practical solutions, this work advances the state of SFT. The source code will be available at https://github.com/yongliang-wu/DFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。