用信息散度统一多种对齐方法，提升语言模型与人类偏好的一致性。
$f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization
- 通过最小化f-散度统一DPO、EXO等对齐算法
- 在AlpacaEval 2等四个基准上性能优于现有方法
- 提供可调的正则化机制，适合研究对齐理论的学者
偏好优化近年来取得显著进展，已发展出多种对齐语言模型与人类偏好的方法。本文提出f-散度偏好优化（f-PO），一种新框架，通过最小化优化策略与最优策略之间的f-散度，涵盖使用不同散度的广泛对齐方法。该方法统一了包括DPO和EXO在内的先前算法，并可通过选择不同的f-散度生成新变体。我们提供了f-PO的理论分析，并在多个前沿语言模型上使用基准数据集进行了广泛实验。结果表明，f-PO在各类任务中均表现出色，在AlpacaEval 2、Arena-Hard、MT-Bench和Open LLM Leaderboard v2等主流基准上优于现有方法。此外，我们还进行了消融研究，探讨不同f-散度的影响，揭示了离线偏好优化中正则化与性能之间的权衡。本工作为语言模型对齐领域贡献了实用算法与理论理解。代码已开源：https://github.com/MinkaiXu/fPO。
原文摘要 · Abstract (English)
Preference optimization has made significant progress recently, with numerous methods developed to align language models with human preferences. This paper introduces $f$-divergence Preference Optimization ($f$-PO), a novel framework that generalizes and extends existing approaches. $f$-PO minimizes $f$-divergences between the optimized policy and the optimal policy, encompassing a broad family of alignment methods using various divergences. Our approach unifies previous algorithms like DPO and EXO, while offering new variants through different choices of $f$-divergences. We provide theoretical analysis of $f$-PO's properties and conduct extensive experiments on state-of-the-art language models using benchmark datasets. Results demonstrate $f$-PO's effectiveness across various tasks, achieving superior performance compared to existing methods on popular benchmarks such as AlpacaEval 2, Arena-Hard, MT-Bench, and Open LLM Leaderboard v2. Additionally, we present ablation studies exploring the impact of different $f$-divergences, offering insights into the trade-offs between regularization and performance in offline preference optimization. Our work contributes both practical algorithms and theoretical understanding to the field of language model alignment. Code is available at https://github.com/MinkaiXu/fPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。