解决多目标强化学习中奖励欺骗问题，让多个目标均衡优化。
MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problems
- 通过方差自适应重加权，自动调节多目标奖励函数贡献。
- 在4个任务中稳定提升性能，避免单一目标过度优化。
- 适合需要平衡多个目标的RL应用，如翻译、控制任务。
当具备准确奖励模型时，分组相对策略优化（GRPO）表现优异。然而，在多数真实任务中，可靠奖励模型难以获得。本文聚焦多目标场景，发现GRPO易受奖励欺骗影响，仅优化单一目标而忽略其他目标。为此，提出MO-GRPO，通过简单归一化方法自动根据各奖励函数值的方差调整其权重。理论上证明，MO-GRPO可确保所有奖励函数对损失函数贡献均等，同时保持偏好顺序，无需手动调校奖励尺度。实验在四个领域验证：(i) 多臂老虎机，(ii) 模拟控制任务（Mo-Gymnasium），(iii) WMT基准上的机器翻译（En-Ja, En-Zh），(iv) 指令遵循任务。结果表明，MO-GRPO通过均衡分配奖励各组件的相关性，实现稳定学习，优于GRPO，展现出在多目标强化学习中的良好前景。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) has been shown to be an effective algorithm when an accurate reward model is available. However, such a highly reliable reward model is not available in many real-world tasks. In this paper, we particularly focus on multi-objective settings, in which we identify that GRPO is vulnerable to reward hacking, optimizing only one of the objectives at the cost of the others. To address this issue, we propose MO-GRPO, an extension of GRPO with a simple normalization method to reweight the reward functions automatically according to the variances of their values. We first show analytically that MO-GRPO ensures that all reward functions contribute evenly to the loss function while preserving the order of preferences, eliminating the need for manual tuning of the reward functions' scales. Then, we evaluate MO-GRPO experimentally in four domains: (i) the multi-armed bandits problem, (ii) simulated control task (Mo-Gymnasium), (iii) machine translation tasks on the WMT benchmark (En-Ja, En-Zh), and (iv) instruction following task. MO-GRPO achieves stable learning by evenly distributing correlations among the components of rewards, outperforming GRPO, showing MO-GRPO to be a promising algorithm for multi-objective reinforcement learning problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。