改进大模型训练的奖励分配机制，提升推理准确性与稳定性。
Future-KL Regularized GRPO: Process-Level Credit Assignment from $f$-Divergence Regularization
- 引入未来KL正则化，捕捉局部惩罚遗漏的梯度信号。
- 在数学推理任务中，pass@16指标显著提升，熵更高且策略漂移更小。
- 无需评论器或额外推理，适合大规模语言模型后训练场景。
组相对策略优化（GRPO）广泛用于无评论器的大语言模型后训练，但其通常采用局部令牌级的KL正则化。我们发现这会遗漏自回归KL正则化带来的策略梯度信号。与标准KL正则化强化学习目标不同，GRPO的组归一化导致提示级效用非线性，对于二值验证器奖励，该效用为 $2\arcsin\sqrt p$。因此，奖励与KL不能在归一化前融合，否则会改变隐含目标。我们推导了带有令牌级 $f$-散度正则化的GRPO式目标的在线策略梯度。奖励项恢复标准化的GRPO优势，而正则项包含被局部KL损失忽略的因果未来正则化回报。对逆KL情况，可得简单修正：在优势构造后添加每令牌对数比的逆累积和。所提方法即未来KL正则化策略优化（FRPO），无需评论器或额外模型推理。在数学推理任务中，FRPO在主设置下提升pass@16，同时保持更高熵和更低策略漂移。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) is widely used for critic-free Large Language Model (LLM) post-training, but its KL regularization is usually implemented as a local loss-side token penalty. We show that this misses the policy-gradient signal induced by autoregressive KL regularization. Unlike standard KL-regularized Reinforcement Learning (RL) objectives, GRPO's group normalization induces a non-linear prompt-level utility; for binary verifier rewards, this utility is $2\arcsin\sqrt p$. As a result, reward and KL cannot be fused before normalization without changing the implicit objective. We derive the on-policy gradient of GRPO-style objectives with token-wise $f$-divergence regularization. The reward term recovers the standardized GRPO advantage, while the regularizer term includes a causal future-regularization return-to-go omitted by local KL losses. For reverse KL, this yields a simple future KL correction: add a reverse cumulative sum of per-token log ratios after advantage construction. The resulting method, Future-KL Regularized Policy Optimization (FRPO), requires no critic or extra model passes. On mathematical reasoning tasks, FRPO improves pass@16 in our main large-model setting while maintaining higher entropy and lower policy drift than conventional loss-side KL baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。