用最大边际似然优化，让大模型更稳定地学习人类偏好。
Offline Preference Optimization via Maximum Marginal Likelihood Estimation
- 将偏好对作为样本，直接最大化优选文本的边际似然。
- 在135M到8B参数模型上，对超参数更鲁棒且效果更优。
- 无需奖励模型和熵正则化，适合追求稳定训练的研究者。
对齐大语言模型与人类偏好至关重要，但传统的基于人类反馈强化学习（RLHF）方法通常复杂且不稳定。本文提出一种新方法——基于最大边际似然（MML）的偏好优化（MMPO），通过将偏好对视为样本，最大化优选文本的边际对数似然，无需显式奖励模型或熵最大化。理论上证明，MMPO隐式实现偏好优化，生成加权梯度，自然提升优选响应的权重。在135M至8B参数的多个模型上，实验表明：1）相对于超参数β更具稳定性；2）实现竞争性或更优的偏好对齐，同时更好保留基础模型的语言能力。消融实验进一步验证，性能提升源于梯度更新中隐含的偏好优化机制。
原文摘要 · Abstract (English)
Aligning Large Language Models (LLMs) with human preferences is crucial, but standard methods like Reinforcement Learning from Human Feedback (RLHF) are often complex and unstable. In this work, we propose a new, simpler approach that recasts alignment through the lens of Maximum Marginal Likelihood (MML) estimation. Our new MML based Preference Optimization (MMPO) maximizes the marginal log-likelihood of a preferred text output, using the preference pair as samples for approximation, and forgoes the need for both an explicit reward model and entropy maximization. We theoretically demonstrate that MMPO implicitly performs preference optimization, producing a weighted gradient that naturally up-weights chosen responses over rejected ones. Across models ranging from 135M to 8B parameters, we empirically show that MMPO: 1) is more stable with respect to the hyperparameter $β$ compared to alternative baselines, and 2) achieves competitive or superior preference alignment while better preserving the base model's general language capabilities. Through a series of ablation experiments, we show that this improved performance is indeed attributable to MMPO's implicit preference optimization within the gradient updates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。