GRPO奖励机制存在误导性优势，SIGNBALANCE可消除该问题。
Spurious Advantage Hidden in GRPO
- 用全局符号+零均值重缩放，消除组内奖励的虚假优势
- 在封闭答案任务中显著提升性能，避免模型靠猜得分
- 适用于数学推理与搜索代理，尤其适合小候选集场景
Group Relative Policy Optimization (GRPO) 在具有可验证奖励的强化学习中广泛应用，其优势估计器基于组内奖励统计为每条轨迹分配一个数值。在常见情况下，该数值会奖励通过推理得出正确答案的轨迹。然而，当轨迹通过猜测恰好命中正确答案时，同一公式仍赋予高数值，形成隐蔽的虚假优势。该现象出现在三类场景：候选集较小的封闭答案任务；包含封闭子任务的开放答案任务；以及预算充足、通向同一答案路径众多的搜索代理。这会导致策略偏向猜测行为。为此提出 SIGNBALANCE，其优势幅度不依赖于组内组成：保留验证器符号，采用全局尺度，并通过每类停止梯度重缩放实现零均值平衡。在不同规模的数学与搜索代理基准测试中，SIGNBALANCE 在开放答案数学任务上与 GRPO 性能相当，在封闭答案数学和搜索代理任务上表现更优。代码将公开。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) is widely studied for reinforcement learning with verifiable rewards, where its advantage estimator assigns each rollout a magnitude from within-group reward statistics. In the common case, this magnitude rewards rollouts that reach the correct answer through reasoning. Yet, an overlooked case shares the same surface: a rollout may land on it by guessing, and the formula still assigns a high magnitude, which we identify as the spurious advantage. This arises in three cases: bounded-answer tasks with a small candidate set; open-answer sets hosting bounded sub-cases; and search agents whose budget opens many paths to the same answer. In all three, this misleads the policy toward guess-like behaviors. We propose SIGNBALANCE, whose magnitude is composition-free: it keeps the verifier sign, uses a global scale, and restores zero-mean balance via a stop-gradient per-class rescaling. Across math and search agent benchmarks at different scales, SIGNBALANCE matches GRPO on open-answer math and improves on bounded-answer math and search agents. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。