提出新方法防止强化学习中奖励欺骗，既能防作弊又不阻碍正常进步。
Modification-Considering Value Learning for Reward Hacking Mitigation in RL
- 用新策略评估每个数据是否该加入训练，只保留不会降低表现的。
- 在4个网格世界和3个控制任务中有效抑制奖励欺骗。
- 适合关注安全强化学习的研究者与工程师使用。
强化学习代理可能利用错误设定的奖励信号，获得高表面回报却偏离原始目标，这种现象称为奖励欺骗。现有实用防御方法通常限制策略更新以保持在已知安全参考附近，导致抑制作弊与允许合理改进之间的矛盾。本文提出修正考虑的价值学习（MCVL），实现基于标准价值函数的当前效用优化。MCVL封装离策略学习器，将每个新样本视为潜在修改：预测包含与不包含该样本的两条训练路径，并用冻结的自举回报估计器（基于学习到的奖励模型与价值函数）评分。仅当包含该样本不会降低评分时才接受。我们形式化了该过滤机制安全且具有包容性的条件，并在DDQN和TD3上实现MCVL。在四个安全相关网格世界及三个含不同欺骗机制的修改版MuJoCo连续控制任务中，MCVL有效缓解奖励欺骗，同时持续提升原始目标性能。
原文摘要 · Abstract (English)
Reinforcement learning agents can exploit misspecified reward signals to achieve high apparent returns while failing on the intended objective, a failure mode known as reward hacking. Existing practical defenses typically constrain policy updates to stay near a known safe reference, creating a tension between suppressing hacking and permitting legitimate improvement. We propose Modification-Considering Value Learning (MCVL), which operationalizes the theoretical idea of current utility optimization for standard value-based RL. MCVL wraps an off-policy learner and treats each incoming transition as a candidate modification: it forecasts two training paths, one that includes the transition and one that does not, and scores both with a frozen bootstrapped-return estimator derived from a learned reward model and value function. The transition is admitted only if inclusion does not decrease the score. We formalize conditions under which this filtering is both safe and permissive, and instantiate MCVL with DDQN and TD3. Across four safety-relevant gridworlds and three modified MuJoCo continuous-control tasks with diverse hacking mechanisms, MCVL mitigates reward hacking while continuing to improve the intended objective. Project website: ktolnos.github.io/mcvl/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。