用对抗样本训练奖励模型，提升其抗欺骗能力。
Adversarial Training of Reward Models
- 通过强化学习自动生成低质量但高奖励的对抗样本。
- 在Nemotron 340B等模型上显著提升鲁棒性，减少奖励劫持。
- 适合关注大模型对齐与安全性的研究者使用。
奖励建模已成为语言模型可扩展对齐的有前景方法。然而，当前奖励模型（RMs）往往缺乏鲁棒性，对低质量、分布外（OOD）样本给予过高奖励，导致奖励劫持——策略利用非预期捷径最大化奖励，破坏对齐效果。为此，我们提出Adv-RM，一种新型对抗训练框架，可自动识别对抗样本：即从目标奖励模型获得高奖励但实际为分布外、低质量的响应。通过强化学习训练一个策略生成这些对抗样本，有效暴露大型先进奖励模型（如Nemotron 340B RM）的漏洞。将此类对抗样本纳入奖励训练过程后，显著提升了奖励模型的鲁棒性，缓解了奖励劫持问题，并改善了下游RLHF性能。实验表明，Adv-RM显著优于传统训练方法，在合成与真实数据设置下均提升稳定性，支持更有效的强化学习人类反馈训练。
原文摘要 · Abstract (English)
Reward modeling has emerged as a promising approach for the scalable alignment of language models. However, contemporary reward models (RMs) often lack robustness, awarding high rewards to low-quality, out-of-distribution (OOD) samples. This can lead to reward hacking, where policies exploit unintended shortcuts to maximize rewards, undermining alignment. To address this challenge, we introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples -- responses that receive high rewards from the target RM but are OOD and of low quality. By leveraging reinforcement learning, Adv-RM trains a policy to generate adversarial examples that reliably expose vulnerabilities in large state-of-the-art reward models such as Nemotron 340B RM. Incorporating these adversarial examples into the reward training process improves the robustness of RMs, mitigating reward hacking and enhancing downstream performance in RLHF. We demonstrate that Adv-RM significantly outperforms conventional RM training, increasing stability and enabling more effective RLHF training in both synthetic and real-data settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。