提出新方法训练更鲁棒的奖励模型，避免因文本长度等无关因素误导。
RRM: Robust Reward Model Training Mitigates Reward Hacking
- 构建因果框架，让奖励模型忽略长度、格式等干扰因素。
- 在RewardBench上将准确率从80.61%提升至84.15%。
- 适合关注大模型对齐与奖励模型可信度的研究者。
奖励模型（RMs）在对齐大语言模型（LLMs）与人类偏好中起关键作用。然而，传统基于特定提示响应对的训练方式难以区分提示相关偏好与提示无关的特征（如响应长度和格式）。本文揭示了当前训练方法的根本缺陷：当判断偏好时，奖励模型无法有效区分上下文信号与无关干扰。为此，我们提出一种因果框架，学习独立于这些干扰的偏好，并设计了一种新颖的数据增强技术以消除干扰因素。大量实验表明，该方法成功过滤了不良特征，得到更鲁棒的奖励模型（RRM）。RRM在使用Gemma-2-9b-it训练的成对奖励模型上，在RewardBench上的准确率从80.61%提升至84.15%。此外，我们使用RM和RRM分别训练了两个DPO策略，结果显示RRM显著提升对齐效果：MT-Bench得分从7.27提升至8.31，AlpacaEval-2中长度控制胜率从33.46%提升至52.49%。
原文摘要 · Abstract (English)
Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. However, traditional RM training, which relies on response pairs tied to specific prompts, struggles to disentangle prompt-driven preferences from prompt-independent artifacts, such as response length and format. In this work, we expose a fundamental limitation of current RM training methods, where RMs fail to effectively distinguish between contextual signals and irrelevant artifacts when determining preferences. To address this, we introduce a causal framework that learns preferences independent of these artifacts and propose a novel data augmentation technique designed to eliminate them. Extensive experiments show that our approach successfully filters out undesirable artifacts, yielding a more robust reward model (RRM). Our RRM improves the performance of a pairwise reward model trained on Gemma-2-9b-it, on RewardBench, increasing accuracy from 80.61% to 84.15%. Additionally, we train two DPO policies using both the RM and RRM, demonstrating that the RRM significantly enhances DPO-aligned policies, improving MT-Bench scores from 7.27 to 8.31 and length-controlled win-rates in AlpacaEval-2 from 33.46% to 52.49%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。