用先验奖励知识优化大模型偏好，提升对齐效果且无需调参。
MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledge
- 将先验奖励融入最大后验目标，改进偏好学习机制。
- 在多个基准上实现一致性能提升，不增加计算开销。
- 可作为插件兼容DPO系列方法，适合需要高效对齐的场景。
随着大语言模型时代的到来，偏好优化（PO）已成为对齐模型与人类偏好、提升性能的核心方法。本文提出最大后验偏好优化（MaPPO），一种将先验奖励知识显式融入优化目标的偏好学习方法。基于直接偏好优化（DPO）及其变体将偏好学习视为最大似然估计的范式，MaPPO将先验奖励估计整合进一个规范的最大后验（MAP）目标中。这不仅推广了DPO及其变体，还通过缓解响应二元分类的过度简化问题提升了对齐效果。此外，MaPPO不引入额外超参数，支持离线与在线两种设置。它还可作为插件用于DPO系列方法，包括广泛使用的SimPO、IPO和CPO，带来稳定提升。在三种标准基准（MT-Bench、AlpacaEval 2.0、Arena-Hard）上，对不同规模和系列模型的大量实验表明，其在不牺牲计算效率的前提下实现了持续的对齐性能提升。
原文摘要 · Abstract (English)
As the era of large language models (LLMs) unfolds, Preference Optimization (PO) methods have become a central approach to aligning LLMs with human preferences and improving performance. We propose Maximum a Posteriori Preference Optimization (MaPPO), a methodology for learning from preferences that explicitly incorporates prior reward knowledge into the optimization objective. Building on the paradigm employed by Direct Preference Optimization (DPO) and its variants of treating preference learning as a Maximum Likelihood Estimation (MLE) problem, MaPPO integrates prior reward estimates into a principled Maximum a Posteriori (MaP) objective. This not only generalizes DPO and its variants, but also enhances alignment by mitigating the oversimplified binary classification of responses. Additionally, MaPPO introduces no additional hyperparameters, and supports preference optimization in both offline and online settings. In addition, MaPPO can be used as a plugin for DPO variants, including widely used SimPO, IPO and CPO, and produce consistent improvements. Extensive empirical evaluations of different model sizes and model series on three standard benchmarks (MT-Bench, AlpacaEval 2.0, and Arena-Hard) demonstrate consistent improvements in alignment performance without sacrificing computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。