用概率框架统一优化语言模型偏好，无需强化学习且性能提升3%~10%
PIPA: Preference Alignment as Prior-Informed Statistical Estimation
- 将偏好对齐建模为带先验约束的最大似然估计
- 在GSM8K和MATH上提升3%~10%，不增加训练成本
- 兼容成对/非成对数据，适合无强化学习场景
离线偏好对齐方法如直接偏好优化（DPO）因其高效与简洁而受到青睐，无需昂贵的强化学习。尽管已有多种针对不同数据设置的离线算法，但缺乏统一理解。本文提出先验信息引导的偏好对齐（PIPA），一个无需强化学习的统一概率框架，将语言模型偏好对齐建模为带有先验约束的最大似然估计（MLE）问题。该方法可有效处理成对与非成对数据，以及答案级和步骤级标注。我们证明DPO与KTO是该框架下不同先验约束的特例。通过整合多种先验信息，我们开发了PIPA-M与PIPA-N两种变体。两者在所有配置下于GSM8K与MATH基准上均实现3%~10%的性能提升，且相比现有算法不增加训练或计算开销。
原文摘要 · Abstract (English)
Offline preference alignment for language models such as Direct Preference Optimization (DPO) is favored for its effectiveness and simplicity, eliminating the need for costly reinforcement learning. Various offline algorithms have been developed for different data settings, yet they lack a unified understanding. In this study, we introduce Pior-Informed Preference Alignment (PIPA), a unified, RL-free probabilistic framework that formulates language model preference alignment as a Maximum Likelihood Estimation (MLE) problem with prior constraints. This method effectively accommodates both paired and unpaired data, as well as answer and step-level annotations. We illustrate that DPO and KTO are special cases with different prior constraints within our framework. By integrating different types of prior information, we developed two variations of PIPA: PIPA-M and PIPA-N. Both algorithms demonstrate a $3\sim10\%$ performance enhancement on the GSM8K and MATH benchmarks across all configurations, achieving these gains without additional training or computational costs compared to existing algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。