解决大模型强化学习中的奖励噪声问题，提升训练稳定性与准确性。
Noise-corrected GRPO: From Noisy Rewards to Unbiased Gradients
- 将奖励错误建模为伯努利噪声，通过概率估计进行偏差修正。
- 在数学任务上准确率提升6.7个百分点，代码任务提升1.5个百分点。
- 适合需要高可靠性奖励信号的现实场景部署，如对齐大模型。
基于人类反馈的强化学习（RLHF）或可验证奖励（RLVR）是对齐大语言模型或构建当前最优推理模型的标准范式，但对不一致或错误奖励带来的噪声高度敏感。然而，此类噪声与广泛使用的基于分组的策略优化方法之间的交互仍缺乏研究。本文提出一种抗噪声的分组相对策略优化（GRPO）及改进版Done Right GRPO（Dr.GRPO）框架，显式将奖励污染建模为伯努利噪声。该方法在估计奖励翻转概率后进行噪声校正，从而获得无偏梯度估计。理论分析表明，分组方法天然能缓解个体层面的噪声，而我们的校正策略进一步增强了这种鲁棒性。实验显示，在数学和代码任务中应用该噪声校正后，性能持续提升：在真实奖励模型条件下，数学任务准确率最高提升6.7个百分点，代码任务提升1.5个百分点。本工作将监督学习中的标签噪声纠正思想引入现代RLHF，兼具理论深度与实际应用价值。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) or verifiable rewards (RLVR), the standard paradigm for aligning LLMs or building recent SOTA reasoning models, is highly sensitive to noise from inconsistent or erroneous rewards. Yet, the interaction between such noise and widely used group-based policy optimization methods remains underexplored. We introduce a noise-robust Group Relative Policy Optimization (GRPO) and Done Right GRPO (Dr.GRPO) framework that explicitly models reward corruption as Bernoulli noise. Our method applies noise correction after estimating reward flip probabilities to debias the learning signal, yielding provably unbiased gradient estimates. Theoretical analysis shows that group-based methods inherently mitigate individual-level noise, and our correction strategy amplifies this robustness. Empirically, we observe consistent improvements across math and code tasks when applying our noise correction to standard reward model usage, with particular gains of up to 6.7 percentage points in accuracy on math tasks and 1.5 on code tasks under realistic reward model conditions. This work bridges label-noise correction from supervised learning with modern RLHF, offering both theoretical insights and a practical algorithm for noisy real-world deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。