解决大模型强化学习中关键梯度失控问题，提升训练稳定性和泛化能力。
All-Quadrant Bounded Clipping GRPO: Closing the Unbounded Blind Spot for Stable and Generalizable Training
- 在四个象限统一施加裁剪，防止负优势下高概率比导致的梯度爆炸。
- 在数学推理任务上达到最高通过率（Pass@64）且熵值更高，更稳定。
- 适合追求高稳定性与泛化性的大模型强化学习研究者使用。
Group Relative Policy Optimization (GRPO) 在大语言模型的强化学习中广受欢迎，但其继承了 PPO 的逐标记裁剪机制，并将逐标记优势替换为单一序列级优势。通过在（似然比，优势）空间的四象限分析，我们发现该组合在负优势与高似然比并存的第四象限（Q4）存在结构上无界的问题，少数高比值标记可能遭受极大抑制性更新，导致熵崩溃和推理边界变窄。为此，我们提出全象限有界裁剪的 GRPO（ABC-GRPO），通过符号依赖的边界对所有四象限实施无条件裁剪。该方法在乘以优势前裁剪似然比，在 Q2 引入信任区域下限，在 Q4 设置上限——其负优势分支与双裁剪 PPO 一致，从而确保每个步骤的策略偏移均有界。在 Qwen3 基础模型的数学推理任务上，ABC-GRPO 达到最高平均得分（Avg@64）和通过率（Pass@64），显著优于 GRPO、SAPO 和双裁剪 PPO，与最强基线 DAPO 相当，同时保持更高熵值；性能提升可迁移至 MATH-500 和跨领域代码生成（HumanEval）。消融实验确认 Q4 是主要盲点。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has emerged as a popular algorithm for reinforcement learning with large language models (LLMs). However, GRPO inherits PPO's token-level clipping while replacing token-level advantages with a single sequence-level advantage. Through a four-quadrant analysis of the (likelihood-ratio, advantage) space, we show that this combination leaves one quadrant -- negative advantage combined with an increased likelihood ratio (Q4) -- structurally unbounded, so that a few high-ratio tokens can receive very large suppressive updates that collapse entropy and narrow the reasoning boundary. To address this, we propose All-Quadrant Bounded Clipping GRPO (ABC-GRPO), which applies unconditional clipping in all four quadrants through sign-dependent boundaries. ABC-GRPO clips the likelihood ratio before multiplying by the advantage, adding a trust-region floor in Q2 and a cap in Q4 -- its negative-advantage branch coinciding with dual-clip PPO -- to yield bounded per-step policy displacement in every quadrant. On mathematical reasoning with Qwen3 base models, ABC-GRPO attains the highest Avg@64 and Pass@64: it is statistically superior to GRPO, SAPO, and dual-clip PPO and competitive with the strongest baseline (DAPO), while maintaining substantially higher entropy; the gains transfer to MATH-500 and to out-of-domain code (HumanEval). Ablations isolate Q4 as the dominant blind spot.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。