通过梯度范数控制优化稳定性，提升大模型推理的泛化能力
Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
- 基于梯度范数设计令牌加权策略，抑制过激更新
- 在数学推理等任务上显著优于传统GRPO，梯度轨迹更平滑
- 适合追求稳定训练和泛化性能的RLVR应用
基于可验证奖励的强化学习（RLVR）已成为提升大语言模型推理能力的有效路径，其中组相对策略优化（GRPO）被广泛使用。然而，RLVR训练通常对泛化能力控制有限。本文从鲁棒性视角重新审视GRPO，发现泛化损失可被经验损失与梯度范数构成的尖锐性代理所上界。基于此，提出尖锐性引导的GRPO（GRPO-SG），一种简单的令牌加权变体，通过降低可能导致过大梯度的令牌权重，减少尖锐更新并稳定优化过程，从而改善泛化。在数学推理、逻辑谜题及工具增强问答任务上的实验表明，GRPO-SG持续优于GRPO，且梯度范数轨迹更平滑，验证了其作为GRPO在RLVR中简单有效的泛化增强方案。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has become a practical route to improve large language model reasoning, and Group Relative Policy Optimization (GRPO) is a widely used optimizer in this setting. However, RLVR training is typically performed with limited control over generalization. We revisit GRPO through a robustness-based generalization view, where the generalization loss is upper bounded by a combination of the empirical loss and a sharpness surrogate measured by the gradient norm. Building on this perspective, we propose Sharpness-Guided GRPO (GRPO-SG), a simple token-weighted variant of GRPO that downweights tokens likely to cause overly large gradients, reducing sharp updates and stabilizing optimization, thereby improving generalization. Experiments across mathematical reasoning, logic puzzles and tool-augmented question answering show consistent improvements over GRPO, along with smoother gradient-norm trajectories, supporting GRPO-SG as a simple and effective generalization-oriented upgrade to GRPO for RLVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。