MPO通过集合级对比优化，提升语言模型对多候选响应的对齐效果。
Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts
- 用集合级对比替代成对比较，扩展了DPO的适用场景
- 在AlpacaEval2上实现17.5%的长度控制胜率提升
- 适合需要处理多个生成结果的模型对齐任务
直接偏好优化（DPO）已成为利用成对偏好对齐语言模型的流行方法。但在实际后训练流程中，策略生成通常为每个提示产生多个候选响应，由奖励模型评分以指导学习。为此，我们提出多偏好优化（MPO），它是DPO的推广，通过将Bradley-Terry模型扩展至选择集与拒绝集之间的组间比较，实现对完整响应集的优化。为进一步提升学习效果，MPO采用基于偏差的加权机制，强调与均值奖励偏离最大的异常响应，从而诱导自适应课程学习。理论上，MPO在每查询响应数为n时，对齐偏差收敛速率为$\mathcal{O}\left(\frac{1}{\sqrt{n}}\right)$。实验表明，MPO在UltraFeedback基准上达到当前最佳性能，在AlpacaEval2的长度控制胜率上相较最先进基线提升约17.5%，确立了基于偏好的对齐新基准。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) has become a popular approach for aligning language models using pairwise preferences. However, in practical post-training pipelines, on-policy generation typically yields multiple candidate responses per prompt, which are scored by a reward model to guide learning. In this setting, we propose $\textbf{Multi-Preference Optimization (MPO)}$, a generalization of DPO that optimizes over entire sets of responses by extending the Bradley-Terry model to groupwise comparisons between chosen and rejected sets. To further enhance learning, MPO employs deviation-based weighting, which emphasizes outlier responses that deviate most from the mean reward, effectively inducing a self-paced curriculum. We theoretically prove that MPO reduces alignment bias at a rate of $\mathcal{O}\left(\frac{1}{\sqrt{n}}\right)$ with respect to the number of responses per query. Empirically, MPO achieves state-of-the-art performance on the UltraFeedback benchmark and yields up to $\sim 17.5\%$ improvement over the state-of-the-art baseline in length-controlled win rate on AlpacaEval2, establishing a new baseline for preference-based alignment
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。