提出新偏好模型，更好捕捉人类非传递性判断。
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
- 用嵌入空间表示响应，高效建模复杂偏好结构。
- 在RewardBench上超越传统方法，能处理循环偏好。
- 适合需要精细对齐人类价值观的模型训练场景。
建模人类偏好对对齐基础模型与人类价值观至关重要。传统奖励建模方法如Bradley-Terry（BT）模型表达能力有限，难以处理非传递性偏好。本文提出偏好嵌入方法，将响应映射到潜在空间以高效捕捉复杂偏好结构，实现线性查询复杂度。同时引入基于偏好得分的通用偏好优化（GPO），推广了基于人类反馈的强化学习（RLHF）。实验表明，所提出的通用偏好嵌入模型（GPM）在RewardBench基准上持续优于BT模型，且在循环偏好场景中，任何BT模型表现如随机猜测，而本方法有效建模。下游任务如AlpacaEval2.0评估显示，使用GPO和本模型进行语言模型后训练，性能优于BT模型。结果表明该方法有助于更精准地对齐基础模型与细微的人类价值观。代码已公开于https://github.com/general-preference/general-preference-model。
原文摘要 · Abstract (English)
Modeling human preferences is crucial for aligning foundation models with human values. Traditional reward modeling methods, such as the Bradley-Terry (BT) reward model, fall short in expressiveness, particularly in addressing intransitive preferences. In this paper, we introduce preference embedding, an approach that embeds responses into a latent space to capture intricate preference structures efficiently, achieving linear query complexity. Additionally, we propose preference score-based General Preference Optimization (GPO), which generalizes reward-based reinforcement learning from human feedback (RLHF). Experimental results show that our General Preference embedding Model (GPM) consistently outperforms the BT reward model on the RewardBench benchmark and effectively models cyclic preferences where any BT reward model behaves like a random guess. Furthermore, evaluations on downstream tasks such as AlpacaEval2.0, following the language model post-training with GPO and our general preference model, reveal performance improvements over BT models. These findings indicate that our method may enhance the alignment of foundation models with nuanced human values. The code is available at https://github.com/general-preference/general-preference-model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。