统一大模型对齐方法，揭示关键设计选择的影响
Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment
- 提出统一数学框架RPO，整合DPO、IPO等对齐方法
- 实验证明响应数量和奖励模型类型显著影响对齐效果
- 适合研究大模型对齐机制或优化策略的开发者
大型语言模型（LLM）对齐算法的快速发展导致方法体系复杂且碎片化，缺乏对不同方法有效性及其关联性的清晰理解。本文提出奖励感知偏好优化（Reward-Aware Preference Optimization, RPO），一个统一主流偏好优化技术的数学框架，涵盖DPO、IPO、SimPO及REINFORCE（LOO）等方法。RPO提供结构化方式，分离并系统分析优化目标、每提示生成的响应数量、隐式与显式奖励模型使用等设计选择对模型偏好优化的影响。我们还提出一种新的实验设置，可清洁地直接消融这些设计因素。通过在RPO框架内进行大量消融实验，我们揭示了塑造模型对齐的关键因素，为提升大模型对齐效果提供了实用指导。
原文摘要 · Abstract (English)
The rapid development of large language model (LLM) alignment algorithms has resulted in a complex and fragmented landscape, with limited clarity on the effectiveness of different methods and their inter-connections. This paper introduces Reward-Aware Preference Optimization (RPO), a mathematical framework that unifies popular preference optimization techniques in LLM alignment, including DPO, IPO, SimPO, and REINFORCE (LOO), among others. RPO provides a structured approach to disentangle and systematically study the impact of various design choices, such as the optimization objective, the number of responses per prompt, and the use of implicit versus explicit reward models, on LLM preference optimization. We additionally propose a new experimental setup that enables the clean and direct ablation of such design choices. Through an extensive series of ablation studies within the RPO framework, we gain insights into the critical factors shaping model alignment, offering practical guidance on the most effective strategies for improving LLM alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。