用奖励排序替代绝对值，让大模型更好理解人类偏好。
GOPO: Policy Optimization using Ranked Rewards
- 基于奖励排序而非大小进行策略优化，避免数值偏差
- 训练过程更稳定，奖励得分持续更高且收敛更快
- 适合摘要、指令跟随等难以验证奖励的任务
标准的人类反馈强化学习（RLHF）通过成对偏好数据训练奖励模型，再用于策略优化。但现有方法依赖奖励的绝对数值，而奖励模型仅能捕捉相对偏好，在摘要、指令遵循和对话生成等不可验证奖励场景中常导致性能不佳。我们提出分组序数策略优化（GOPO），仅使用奖励排序信息，忽略具体数值。相比已有方法GRPO，GOPO在非可验证奖励场景下表现出：（1）训练与验证奖励轨迹始终更高；（2）多数中间步骤的LLM评分更优；（3）达到相近性能所需训练步数显著减少。我们在多种任务与模型规模上均验证了其一致性提升。
原文摘要 · Abstract (English)
Standard reinforcement learning from human feedback (RLHF) trains a reward model on pairwise preference data and then uses it for policy optimization. However, while reward models are optimized to capture relative preferences, existing policy optimization techniques rely on absolute reward magnitudes during training. In settings where the rewards are non-verifiable such as summarization, instruction following, and chat completion, this misalignment often leads to suboptimal performance. We introduce Group Ordinal Policy Optimization (GOPO), a policy optimization method that uses only the ranking of the rewards and discards their magnitudes. Our rank-based transformation of rewards provides several gains, compared to Group Relative Policy Optimization (GRPO), in settings with non-verifiable rewards: (1) consistently higher training/validation reward trajectories, (2) improved LLM-as-judge evaluations across most intermediate training steps, and (3) reaching a policy of comparable quality in substantially less training steps than GRPO. We demonstrate consistent improvements across a range of tasks and model sizes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。