揭示大模型强化学习中长度偏差与无偏性的根本矛盾。
On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
- 提出f_alpha(L)权重族，统一分析长度加权策略的优劣。
- 证明在标准奖励下，无偏与长度无关不可兼得，存在定量权衡。
- 解释GRPO与Dr. GRPO各占权衡两端，均非完美方案。
Group Relative Policy Optimization (GRPO) 是训练大语言模型推理能力的主流强化学习算法，被 DeepSeek-R1 广泛采用。近期改进 Dr. GRPO（COLM 2025）指出 GRPO 中基于轨迹长度归一化的响应级长度偏差，并移除该归一化，声称新优化器“无偏”。我们证明此声明不完整：在标准结果奖励 + GRPO 框架下，任何基于长度的加权方案都无法同时满足两项性质：(P1) 梯度无偏性——梯度估计为真实策略梯度的无偏估计；(P2) 长度不变性——每条轨迹对梯度的有效贡献与其词元长度无关。GRPO 近似满足 P2 但违反 P1；Dr. GRPO 满足 P1 但违反 P2。我们通过参数族 f_alpha(L) = L^{alpha-1} 完整刻画该权衡谱系，其中 alpha=0 对应 GRPO，alpha=1 对应 Dr. GRPO。定量分析表明，Dr. GRPO 的长度偏差可使更长轨迹的梯度更新主导程度随长度比线性放大。结果揭示：两者均非“正确”方案，而是不可避免的根本权衡之两端。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) is the dominant reinforcement learning algorithm for training reasoning capabilities in large language models, notably adopted by DeepSeek-R1. The recent improvement Dr. GRPO (COLM 2025) identifies the response-level length bias caused by per-trajectory length normalization in GRPO and proposes removing this normalization, claiming the resulting optimizer is "unbiased." We show that this claim is incomplete. Specifically, we establish an impossibility theorem: under the standard outcome reward + GRPO setting, no length-based weighting scheme can simultaneously achieve the following two properties. (P1) Gradient unbiasedness: the gradient estimator is an unbiased estimate of the true policy gradient. (P2) Length invariance: each trajectory's effective contribution to the gradient is independent of its token length. GRPO approximately satisfies P2 but violates P1; Dr. GRPO satisfies P1 but violates P2. We characterize the complete tradeoff spectrum via the parametric family f_alpha(L) = L^{alpha - 1}, where alpha = 0 recovers GRPO, alpha = 1 recovers Dr. GRPO, and provide quantitative analysis showing that Dr. GRPO's length bias can cause longer trajectories to dominate gradient updates by a factor proportional to the length ratio. Our results reveal that neither algorithm is universally "done right"; they occupy opposite ends of a fundamental and unavoidable tradeoff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。