二值奖励下强化学习易梯度枯竭，简单修正显著提升性能
Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works
- 用固定参考符号优势替代均值中心化，解决二值奖励下的梯度消失问题
- 在GSM8K测试集上准确率从28.4%提升至73.8%，提升45.4个百分点
- 适合关注奖励信号有效性与强化学习训练稳定性的研究者
Group Relative Policy Optimization (GRPO) 是基于可验证奖励的强化学习标准算法，但在二值奖励下其组均值中心化的优势会失效。失效表现为梯度枯竭：当组内所有响应均正确或全错误时，中心化优势恰好为零，导致策略无法获得学习信号。我们证明真实退化率始终高于独立同分布伯努利预测，且在已记录的Qwen3.5-9B GSM8K训练中，组大小为4时退化率达0.69。随后我们展示固定参考符号优势 $A=2r-1$ 能通过提高组内至少一个样本成功的概率，实现 pass@$G$ 失败下降。在七个随机种子的GSM8K全测试集上，Sign方法达到73.8%准确率，而标准归一化组均值DrGRPO仅为28.4%，提升45.4个百分点（p<0.0001）。Llama-3.1-8B上方向一致，MATH-500迁移测试中呈正向但统计功效不足。pass@$k$分析表明，主要收益来自搜索压缩而非容量扩展，与近期RLVR上限观察一致。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) is a standard algorithm for reinforcement learning from verifiable rewards, but its group-mean-centered advantage can fail under binary rewards. The failure mode is gradient starvation: when every response in a group is correct or every response is wrong, the centered advantage is exactly zero and the policy receives no learning signal. We prove that the true degeneracy rate always exceeds the i.i.d. Bernoulli prediction by Jensen's inequality, and observe a 0.69 degeneracy rate at group size four in logged Qwen3.5-9B GSM8K training. We then show that the fixed-reference Sign advantage, $A=2r-1$, performs pass@$G$ failure descent by increasing the probability that at least one sample in the group succeeds. On the full GSM8K test set across seven seeds, Sign reaches 73.8% accuracy versus 28.4% for standard normalized group-mean DrGRPO at group size four, a 45.4 point gain with $p<0.0001$. The effect is directionally consistent on Llama-3.1-8B and positive but underpowered on a MATH-500 transfer check. Pass@$k$ analysis indicates that the main benefit is search compression rather than large capacity expansion, aligning the empirical gains with recent RLVR ceiling observations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。