提出首个与真实人类偏好对齐的奖励模型评估基准
How to Evaluate Reward Models for RLHF
- 用代理任务预测奖励模型在强化学习中的表现
- 12个领域测12项指标，发现关键相关性特征
- 适合训练和评估大模型对齐效果的研究者使用
本文提出一种新的奖励模型评估基准，用于衡量其通过强化学习从人类反馈（RLHF）生成优质语言模型的能力。传统方法需完整运行RLHF流程并测试下游大模型性能，成本极高。为此，我们构建了基于代理任务的预测模型，涵盖大规模人类偏好数据集和可验证正确性数据集，在12个领域测量12项指标。为确定哪些指标与真实RLHF结果最相关，我们在一个大规模众包人类偏好平台上开展端到端的RLHF实验，以真实人类偏好作为基准。最终，我们整理出「偏好代理评估」（PPE），这是首个明确关联后RLHF实际人类偏好表现的奖励模型基准，已开源供公众使用和进一步开发。代码与评估见 https://github.com/lmarena/PPE。
原文摘要 · Abstract (English)
We introduce a new benchmark for reward models that quantifies their ability to produce strong language models through RLHF (Reinforcement Learning from Human Feedback). The gold-standard approach is to run a full RLHF training pipeline and directly probe downstream LLM performance. However, this process is prohibitively expensive. To address this, we build a predictive model of downstream LLM performance by evaluating the reward model on proxy tasks. These proxy tasks consist of a large-scale human preference and a verifiable correctness preference dataset, in which we measure 12 metrics across 12 domains. To investigate which reward model metrics are most correlated to gold-standard RLHF outcomes, we launch an end-to-end RLHF experiment on a large-scale crowdsourced human preference platform to view real reward model downstream performance as ground truth. Ultimately, we compile our data and findings into Preference Proxy Evaluations (PPE), the first reward model benchmark explicitly linked to post-RLHF real-world human preference performance, which we open-source for public use and further development. Our code and evaluations can be found at https://github.com/lmarena/PPE .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。