解释了GRPO为何需要标准化,揭示了自适应梯度的内在机制。
Why GRPO Needs Normalization: A Local-Curvature Perspective on Adaptive Gradients
- 从局部曲率视角分析,标准差归一化实现自适应梯度。
- 理论证明其收敛速度优于无归一化的REINFORCE,提升幅度与奖励方差相关。
- 在GSM8K和MATH上发现三阶段训练规律,适合研究无评价值强化学习者。
强化学习已成为语言模型推理的关键驱动力。在众多算法中,组相对策略优化(GRPO)已成为标准方法,通过每提示基线和方差归一化避免使用评价器。然而,这种归一化为何有效、何时有效仍不明确。本文从序列级策略梯度的局部曲率出发,揭示标准差归一化实现了自适应梯度。理论上,在温和条件下,GRPO的收敛速度严格优于未归一化的REINFORCE,增益由各提示在迭代中的平均内部奖励标准差决定。实验在GSM8K和MATH基准上揭示了三个受特征正交性与奖励方差相互作用支配的训练阶段:(I) 早期加速阶段,高方差与正交性有利于自适应缩放;(II) 相对稳定的过渡阶段;(III) 晚期阶段,正交性丧失限制进一步提升。结果为标准差归一化在GRPO中的作用提供了原则性解释,并为无评价值强化学习算法设计提供更广泛洞见。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a key driver of language model reasoning. Among RL algorithms, Group Relative Policy Optimization (GRPO) is the de facto standard, avoiding the need for a critic by using per-prompt baselines and variance normalization. Yet why and when this normalization helps remains unclear. In this work, we provide an explanation through the lens of local curvature of the sequence-level policy gradient: standard deviation normalization implements an adaptive gradient. Theoretically, under mild conditions, GRPO enjoys a strictly improved convergence rate over unnormalized REINFORCE, with gains characterized by the average within-prompt reward standard deviation across prompts and iterations. Empirically, our analysis on GSM8K and MATH benchmarks reveals three distinct training phases governed by the interplay between feature orthogonality and reward variance: (I) an early acceleration phase where high variance and orthogonality favor adaptive scaling; (II) a relatively stable transition phase; and (III) a late-stage regime where the loss of orthogonality limits further gains. Together, these results provide a principled account of when std normalization helps in GRPO, and offer broader insights into the design of critic-free RL algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。