提出统一的强化学习算法框架，提升大模型推理稳定性与准确率。
On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning
- 从理论推导出KL正则化策略梯度的统一形式，明确各变体本质
- 在数学推理任务上实现最高52%准确率，比基线提升6个百分点
- 适合需要稳定高精度推理的大模型训练场景
策略梯度算法已成功用于增强大语言模型（LLMs）的推理能力。尽管KL正则化被广泛使用，但其设计空间——包括前向/反向KL、归一化与否、估计器类型（k1/k2/k3）——在文献中分散且常与非策略估计交织。本文聚焦于非策略设定下：为使优化的代理目标精确对应预期的KL正则化目标，每种KL变体需采用何种权重？我们通过统一推导提出正则化策略梯度（RPG）视角。RPG（i）统一了归一化与非归一化KL变体，证明广泛使用的k3惩罚项即为非归一化KL；（ii）给出了带停止梯度的REINFORCE式损失与全可微代理梯度等价的条件；（iii）识别并修正了GRPO中KL项的非策略重要性权重不匹配问题；（iv）引入RPG-Style Clip，在RPG-REINFORCE中通过裁剪重要性采样实现大规模稳定非策略训练。在数学推理基准（AIME24, AIME25）上，RPG-REINFORCE配合RPG-Style Clip相比DAPO准确率提升高达+6个百分点。扩展至8K上下文长度后，该方法在AIME25上达到52%准确率，超越官方Qwen3-4B-Instruct模型（47%）。值得注意的是，RPG是稳定且可扩展的LLM推理强化学习算法，其实现依赖于（a）KL校正的目标函数、（b）裁剪重要性采样、（c）迭代参考策略更新机制。
原文摘要 · Abstract (English)
Policy gradient algorithms have been successfully applied to enhance the reasoning capabilities of large language models (LLMs). KL regularization is ubiquitous, yet the design surface, choice of KL direction (forward vs. reverse), normalization (normalized vs. unnormalized), and estimator ($k_1/k_2/k_3$), is scattered across the literature and often intertwined with off-policy estimation. We ask a focused question: under the off-policy setting, what weighting is required for each KL variant so that the surrogate we optimize yields the exact gradient of the intended KL-regularized objective? We answer this with a compact, unified derivation we call the Regularized Policy Gradient (RPG) view. RPG (i) unifies normalized and unnormalized KL variants and shows that the widely-used $k_3$ penalty is exactly the unnormalized KL; (ii) specifies conditions under which REINFORCE-style losses with stop-gradient are gradient-equivalent to fully differentiable surrogates; (iii) identifies and corrects an off-policy importance-weighting mismatch in GRPO's KL term; and (iv) introduces RPG-Style Clip, a clipped-importance-sampling step within RPG-REINFORCE that enables stable, off-policy policy-gradient training at scale. On mathematical reasoning benchmarks (AIME24, AIME25), RPG-REINFORCE with RPG-Style Clip improves accuracy by up to $+6$ absolute percentage points over DAPO. We extend our experiments to 8K context length, and RPG-REINFORCE with RPG-Style Clip achieves 52% accuracy on AIME25, surpassing the official Qwen3-4B-Instruct model (47%). Notably, RPG is a stable and scalable RL algorithm for LLM reasoning, realized via (a) a KL-correct objective, (b) clipped importance sampling, and (c) an iterative reference-policy update scheme. Project Page: https://github.com/complex-reasoning/RPG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。