提出GFT框架，解决大模型微调中奖励稀疏与梯度不稳问题。
GFT: From Imitation to Reward Fine-Tuning with Unbiased Group Advantages and Dynamic Coefficient Rectification
- 通过分组优势学习构建多样化回复，缓解奖励稀疏
- 动态系数修正稳定优化，防止梯度爆炸与熵坍缩
- 适合需要平滑衔接强化学习的高效微调场景
大语言模型通常通过监督微调（SFT）和强化学习（RL）进行后训练，但如何有效融合高效知识注入与强泛化能力仍具挑战。本文通过训练动态分析发现，SFT可被视作一种具有极稀疏隐式奖励和不稳定逆概率加权的策略梯度优化，导致单路径依赖、熵坍缩与梯度爆炸。为此，我们提出统一的后训练框架GFT，包含两个机制：分组优势学习（Group Advantage Learning），通过构建多样响应组并生成归一化对比监督，缓解奖励稀疏；动态系数修正（Dynamic Coefficient Rectification），自适应约束逆概率权重以稳定优化，同时保留高效知识注入。实验表明，GFT持续优于基于SFT的方法，且生成策略更平滑地衔接后续强化学习训练。
原文摘要 · Abstract (English)
Large language models are typically post-trained using supervised fine-tuning (SFT) and reinforcement learning (RL), yet effectively unifying efficient knowledge injection with robust generalization remains challenging. In this work, we provide a training-dynamics analysis showing that SFT can be interpreted as a special case of policy gradient optimization with an extremely sparse implicit reward and unstable inverse-probability weighting, which together lead to single-path dependency, entropy collapse, and gradient explosion. Motivated by this diagnosis, we propose Group Fine-Tuning (GFT), a unified post-training framework that addresses these intrinsic limitations through two mechanisms: Group Advantage Learning, which constructs diverse response groups and derives normalized contrastive supervision to alleviate reward sparsity, and Dynamic Coefficient Rectification, which adaptively bounds inverse-probability weights to stabilize optimization while preserving efficient knowledge injection. Experiments demonstrate that GFT consistently surpasses SFT-based methods and yields policies that integrate more smoothly with subsequent RL training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。