MPPO通过多回复配对优化，提升大模型对齐效率与效果。
MPPO: Multi Pair-wise Preference Optimization for LLMs with Arbitrary Negative Samples
- 采用配对偏好优化思路，利用平均响应似然拟合奖励函数。
- 在MT-Bench和Arena-Hard上均显著优于DPO、ORPO等基线方法。
- 适用于多回复场景，有效利用冗余反馈数据，适合实际应用部署。
大语言模型与人类反馈对齐对发展至关重要。现有偏好优化方法如DPO和KTO虽基于强化学习从人类反馈（RLHF）改进，但源于PPO，需参考模型增加显存开销，且高度依赖大量偏好数据。当前研究多集中于单问题双回复场景，忽视多回复优化，导致数据浪费。本文提出MPPO算法，利用模型响应的平均似然拟合奖励函数，最大化偏好数据利用率。对比点式、配对式和列表式实现发现，配对式表现最优，显著提升模型输出质量。实验表明，MPPO在多个基准测试中表现优异：在MT-Bench上超越DPO、ORPO和SimPO；在Arena-Hard上大幅领先于DPO和ORPO。这些成果凸显了MPPO在偏好优化任务中的显著优势。
原文摘要 · Abstract (English)
Aligning Large Language Models (LLMs) with human feedback is crucial for their development. Existing preference optimization methods such as DPO and KTO, while improved based on Reinforcement Learning from Human Feedback (RLHF), are inherently derived from PPO, requiring a reference model that adds GPU memory resources and relies heavily on abundant preference data. Meanwhile, current preference optimization research mainly targets single-question scenarios with two replies, neglecting optimization with multiple replies, which leads to a waste of data in the application. This study introduces the MPPO algorithm, which leverages the average likelihood of model responses to fit the reward function and maximizes the utilization of preference data. Through a comparison of Point-wise, Pair-wise, and List-wise implementations, we found that the Pair-wise approach achieves the best performance, significantly enhancing the quality of model responses. Experimental results demonstrate MPPO's outstanding performance across various benchmarks. On MT-Bench, MPPO outperforms DPO, ORPO, and SimPO. Notably, on Arena-Hard, MPPO surpasses DPO and ORPO by substantial margins. These achievements underscore the remarkable advantages of MPPO in preference optimization tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。