通过保证优势符号稳定，防止强化学习中因奖励欺骗导致质量下降。
Mitigating Reward Hacking in RLHF via Advantage Sign Robustness
- 利用奖励模型参数扰动推导出符号保真半径，识别并抑制有害更新。
- 在TL;DR和AlpacaFarm上胜率优于基线，有效减少奖励欺骗现象。
- 仅需奖励模型参数和策略生成结果，无需额外数据或多个模型。
在人类反馈强化学习（RLHF）中，奖励模型（RMs）易受奖励欺骗影响：当策略最大化学习到的代理奖励时，真实质量趋于停滞或下降。我们假设奖励欺骗常由优势符号翻转引起——即本应降低不良响应概率的更新反而提升了它。通过在奖励模型参数空间引入对抗性扰动，可推导出一个经认证的符号保真半径，即引发优势符号翻转所需的最小扰动量。基于此，我们提出符号认证策略优化（SignCert-PO），在策略梯度更新中对非鲁棒生成内容进行降权。与以往需多个奖励模型或访问训练数据的方法不同，SignCert-PO轻量且仅在策略优化阶段使用奖励模型参数与在线生成内容。在TL;DR摘要与AlpacaFarm基准测试中，SignCert-PO持续取得更高胜率，并显著降低奖励欺骗。
原文摘要 · Abstract (English)
Reward models (RMs) used in reinforcement learning from human feedback (RLHF) are vulnerable to reward hacking: as the policy maximizes a learned proxy reward, true quality plateaus or degrades. We make the assumption that reward hacking is often caused by flipped advantage signs: instead of reducing the likelihood of a bad response, a flipped sign causes the update to increase it. By considering an adversarial perturbation in the RM parameter space, we can derive a certified sign-preservation radius, which is the smallest perturbation that can flip the advantage sign during policy optimization. Based on this formulation, we propose Sign-Certified Policy Optimization (SignCert-PO), down-weighting non-robust completions in the policy gradient update. Unlike prior approaches that require multiple RMs or access to the RM training data, SignCert-PO is lightweight and operates purely at the policy optimization stage using only the RM parameters and on-policy completions. On TL;DR summarization and AlpacaFarm benchmarks, SignCert-PO consistently achieves a better win rate than baselines and reduces reward hacking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。