GIFT用隐式奖励匹配替代显式奖励最大化，实现更稳定高效的LLM强化学习。
GIFT: Group-Relative Implicit Fine-Tuning Integrates GRPO with DPO and UNA
- 结合分组采样、隐式奖励与优势值误差，自动调节温度参数β。
- 在7B-32B模型上收敛更快，过拟合更少，生成质量更高。
- 适合追求稳定训练和自适应调参的LLM微调研究者。
本文探讨了在大语言模型的在线强化学习中，奖励匹配是否可作为奖励最大化的可行替代方案。提出群相对隐式微调（GIFT），融合GRPO式的分组采样、DPO式的隐式奖励以及UNA式的隐显优势均方误差。通过z-score标准化，消除了DPO隐式奖励中的不可计算分区函数$Z(x)$，并移除了RLHF与RLVR目标中的KL系数$β$。${L}_{\text{GIFT}}$的全局最小化解以闭式表征：精确对应于GRPO/RLHF解族$π^{*}_β(y|x)\proptoπ_{\text{ref}}(y|x)e^{\frac{1}{β}r_ϕ(x,y)}$，其中KL系数$β(x)=\frac{σ_ϕ(x)}{\hatσ_θ(x)}$为依赖提示的、由方差决定的自适应变量。因此，GIFT在保持与GRPO相同参数策略族的同时，将外部调参的标量$β$替换为由奖励分布匹配内生优化的提示自适应$β(x)$。实验表明，在7B-32B骨干模型上，GIFT收敛速度优于GRPO、DAPO与GSPO，且在RLVR任务（GSM8K、MATH、AIME）上过拟合更少，在RLHF任务（AlpacaEval、Arena-Hard）上生成长度控制下的胜率更高。所有证明与详细背景见附录。
原文摘要 · Abstract (English)
This paper investigates whether reward matching is a viable alternative to reward maximization methods for on-policy RL of LLMs. Group-relative Implicit Fine-Tuning (GIFT) is proposed, combining GRPO-style group sampling, DPO-style implicit reward, and UNA-style MSE between implicit and explicit advantages. By applying z-score standardization, the intractable partition function $Z(x)$ in the DPO implicit reward is canceled, and the KL coefficient $β$ is eliminated from the RLHF and RLVR objective. The population minimizers of $\mathcal{L}_{\text{GIFT}}$ are characterized in closed form: they coincide exactly with the GRPO/RLHF solution family $π^{*}_β(y|x)\proptoπ_{\text{ref}}(y|x)e^{\frac{1}βr_ϕ(x,y)}$, with a prompt-dependent, variance-determined KL coefficient $β(x)=\frac{σ_ϕ(x)}{\hatσ_θ(x)}$. GIFT therefore solves the same parametric policy family as GRPO while replacing GRPO's externally tuned scalar $β$ with a prompt-adaptive $β(x)$ optimized endogenously by matching reward distributions. Empirically, on 7B-32B backbones, GIFT converges faster than GRPO, DAPO and GSPO and overfits less on RLVR (GSM8K, MATH, AIME) and produces higher length-controlled win rates on RLHF (AlpacaEval, Arena-Hard). All proofs and detailed background are deferred to the appendix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。