用排序反馈提升大模型对齐效果，比传统两两比较更有效。
Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
- 基于最大似然估计，统一处理多选排序与两两比较反馈
- 在多个模型和场景下，性能均优于现有基线方法
- 适合希望利用更丰富人类反馈的对齐研究者
大语言模型对齐主要依赖两两偏好优化，即标注者从两个回复中选择更优者。该方法虽简单，却忽略了多路比较和前k名排序等更丰富的反馈形式。本文提出排名选择偏好优化（RCPO），通过最大似然估计将偏好优化与（排序）选择建模统一起来。RCPO支持基于效用和基于排名的模型，可包含DPO、SimPO等两两方法作为特例，并为更复杂的反馈格式提供严谨的训练目标。我们以多项式逻辑和Mallows-RMJ为例进行实例化。在Llama-3-8B-Instruct、Gemma-2-9B-it和Mistral-7B-Instruct上，跨分布内与分布外设置的实验表明，RCPO始终优于竞争性基线。结果表明，直接利用排序偏好数据并结合合适的建模方法，能实现更有效的对齐。该框架为将（排序）选择建模融入大模型训练提供了可扩展基础。
原文摘要 · Abstract (English)
Alignment of large language models (LLMs) has predominantly relied on pairwise preference optimization, where annotators select the better of two responses to a prompt. While simple, this approach overlooks the opportunity to learn from richer forms of human feedback, such as multiway comparisons and top-$k$ rankings. We introduce Ranked Choice Preference Optimization (RCPO), a unified framework that bridges preference optimization with (ranked) choice modeling via maximum likelihood estimation. RCPO supports both utility-based and rank-based models, subsumes several pairwise methods (such as DPO and SimPO) as special cases, and provides principled training objectives for richer feedback formats. We instantiate this framework with two representative models (Multinomial Logit and Mallows-RMJ). Experiments on Llama-3-8B-Instruct, Gemma-2-9B-it, and Mistral-7B-Instruct across in-distribution and out-of-distribution settings show that RCPO consistently outperforms competitive baselines. RCPO shows that directly leveraging ranked preference data, combined with the right choice models, yields more effective alignment. It offers an extensible foundation for incorporating (ranked) choice modeling into LLM training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。