新基准测试揭示大模型奖励模型对细微差异敏感度不足
RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style
- 设计新基准评估奖励模型对内容微调与风格偏见的敏感性
- 40个模型平均仅46.6%准确率，低于随机水平
- 适合研究模型对齐与奖励机制优化的学者参考
奖励模型在基于人类反馈的强化学习（RLHF）和推理缩放定律中至关重要，用于引导语言模型对齐并选择最优回复。然而，现有基准多通过对比不同能力模型生成的回答来评估，难以衡量模型对细微内容变化和风格差异的敏感性，导致与策略模型性能相关性低。为此，我们提出RM-Bench，一个专注于评估奖励模型对微妙内容差异与风格偏见抵抗能力的新基准。大量实验表明，RM-Bench与策略模型性能高度相关，可作为选择有效对齐奖励模型的可靠依据。我们在该基准上评估了近40个奖励模型，结果发现即使是顶尖模型，在风格偏见干扰下平均准确率仅为46.6%，低于随机水平（50%），凸显当前奖励模型仍有巨大改进空间。相关代码与数据已开源。
原文摘要 · Abstract (English)
Reward models are critical in techniques like Reinforcement Learning from Human Feedback (RLHF) and Inference Scaling Laws, where they guide language model alignment and select optimal responses. Despite their importance, existing reward model benchmarks often evaluate models by asking them to distinguish between responses generated by models of varying power. However, this approach fails to assess reward models on subtle but critical content changes and variations in style, resulting in a low correlation with policy model performance. To this end, we introduce RM-Bench, a novel benchmark designed to evaluate reward models based on their sensitivity to subtle content differences and resistance to style biases. Extensive experiments demonstrate that RM-Bench strongly correlates with policy model performance, making it a reliable reference for selecting reward models to align language models effectively. We evaluate nearly 40 reward models on RM-Bench. Our results reveal that even state-of-the-art models achieve an average performance of only 46.6%, which falls short of random-level accuracy (50%) when faced with style bias interference. These findings highlight the significant room for improvement in current reward models. Related code and data are available at https://github.com/THU-KEG/RM-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。