用相对排名替代绝对得分,提升大模型推理强化学习的稳定性
From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning
- 将奖励机制从绝对分数转为组内相对排名
- 在多个推理任务上优于传统基线方法
- 适合需要稳定奖励信号的开放生成场景
强化学习已成为提升大语言模型推理能力的核心方法,基于群体的方法(如GRPO)通过利用组内表现差异优化策略。然而,这些方法通常依赖绝对数值奖励,存在固有局限:在可验证任务中,相同组评价导致监督信号稀疏;在开放性场景中,奖励模型评分范围不稳定,影响基于组均值的优势估计。为此,我们提出相对奖励强化学习(RLRR)框架,将奖励设计从绝对评分转向相对排序。同时引入面向群体优化的排序奖励模型,直接生成相对排名。通过将原始评估转化为鲁棒的相对信号,RLRR有效缓解了信号稀疏与奖励不稳问题。实验表明,RLRR在多个推理基准和开放生成任务中均持续优于标准群体基线。
原文摘要 · Abstract (English)
Reinforcement learning has become a cornerstone for enhancing the reasoning capabilities of Large Language Models, where group-based approaches such as GRPO have emerged as efficient paradigms that optimize policies by leveraging intra-group performance differences. However, these methods typically rely on absolute numerical rewards, introducing intrinsic limitations. In verifiable tasks, identical group evaluations often result in sparse supervision, while in open-ended scenarios, the score range instability of reward models undermines advantage estimation based on group means. To address these limitations, we propose Reinforcement Learning with Relative Rewards (RLRR), a framework that shifts reward shaping from absolute scoring to relative ranking. Complementing this framework, we introduce the Ranking Reward Model, a listwise preference model tailored for group-based optimization to directly generate relative rankings. By transforming raw evaluations into robust relative signals, RLRR effectively mitigates signal sparsity and reward instability. Experimental results demonstrate that RLRR yields consistent performance improvements over standard group-based baselines across reasoning benchmarks and open-ended generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。