GRPO虽简化了PPO，但存在奖励分配缺陷导致梯度稀疏。
On the Policy Gradient Foundations of Group Relative Policy Optimization: Credit Assignment, Gradient Sparsity, and Rank Collapse
- 用分组平均奖励替代价值函数，简化策略优化过程
- 实证发现梯度矩阵有效秩始终约等于2，与分组大小无关
- 适用于追求简单高效的推理场景，但不适用于多步逻辑任务
Group Relative Policy Optimization（GRPO）通过使用分组采样轨迹的均值奖励作为基线，摒弃了PPO中的学习型价值函数。本文从策略梯度定理出发，严格推导出GRPO的理论基础，揭示其核心缺陷：在仅依赖输出奖励的情况下，所有轨迹上的每个标记都获得相同的优势值，导致标记级奖励分配坍缩为单一标量。我们证明这一机制会引发梯度稀疏性，并随训练进程加剧。通过对Nemotron-4B/GSM8K数据集上GRPO梯度进行SVD分析，实证表明无论分组大小R ∈ {2, 4, 8}，梯度矩阵的有效秩均≈2。我们将其形式化为源于优势值零和约束的内在秩-2结构，并推导出基线最优的条件。研究结果明确了GRPO简化设计的理论适用边界，指出了奖励分配瓶颈是其在多步推理任务中表现受限的关键原因。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) eliminates the learned critic in PPO by using the mean reward of grouped rollouts as a baseline. We provide a rigorous derivation of GRPO from first principles of the policy gradient theorem, revealing a fundamental credit assignment failure: under output-only reward, every token in a rollout receives identical advantage, collapsing token-level credit to a single scalar. We prove this induces gradient sparsity that intensifies over training, and demonstrate empirically via SVD analysis of GRPO gradients on Nemotron-4B/GSM8K that the gradient matrix has effective rank $\approx$ 2 regardless of group size $R \in \{2, 4, 8\}$. We formalize this as an intrinsic rank-2 structure arising from the zero-sum constraint on advantages and derive conditions under which GRPO's baseline is optimal. Our results characterize when GRPO's simplicity is theoretically justified and identify the credit assignment bottleneck as the key limitation for multi-step reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。