用评分差距提升大模型对齐效率，更快更稳。
Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains
- 引入评分差距信息优化对齐算法，比传统DPO更高效
- 理论证明在准确评分下收敛速度更快，实测性能更优
- 对评分误差不敏感，适合实际应用中的不完美数据
直接偏好优化（DPO）类算法因其仅需成对偏好反馈即可对齐基础模型而备受关注。这类方法无需显式学习奖励模型，但其反馈信息存在质量模糊的问题：难以判断模型对优选/劣选回应的似然变化是积极还是消极信号。本文提出新算法，利用评分差距（rating gap）信息——即优选回应相较于劣选回应的优劣程度——来增强对齐过程。我们证明，在评分差距准确时，新算法可实现比DPO更快的统计收敛速率；同时理论与实证均表明，该方法对评分差距的误差具有鲁棒性。在多种大语言模型和评估基准上，新方法表现显著优于多个DPO风格算法。
原文摘要 · Abstract (English)
The class of direct preference optimization (DPO) algorithms has emerged as a promising approach for solving the alignment problem in foundation models. These algorithms work with very limited feedback in the form of pairwise preferences and fine-tune models to align with these preferences without explicitly learning a reward model. While the form of feedback used by these algorithms makes the data collection process easy and relatively more accurate, its ambiguity in terms of the quality of responses could have negative implications. For example, it is not clear if a decrease (increase) in the likelihood of preferred (dispreferred) responses during the execution of these algorithms could be interpreted as a positive or negative phenomenon. In this paper, we study how to design algorithms that can leverage additional information in the form of rating gap, which informs the learner how much the chosen response is better than the rejected one. We present new algorithms that can achieve faster statistical rates than DPO in presence of accurate rating gap information. Moreover, we theoretically prove and empirically show that the performance of our algorithms is robust to inaccuracy in rating gaps. Finally, we demonstrate the solid performance of our methods in comparison to a number of DPO-style algorithms across a wide range of LLMs and evaluation benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。