DPO存在模型设定错误，新方法AuxDPO能有效修复其缺陷。
Why DPO is a Misspecified Estimator and How to Fix It
- 通过引入辅助变量重构损失函数，修正DPO的统计偏差。
- 在模拟任务和大模型对齐中，性能优于原版DPO。
- 适合关注对齐算法稳定性和理论可靠性的研究者。
直接对齐算法如直接偏好优化（DPO）基于偏好数据进行微调，仅使用监督学习而非两阶段强化学习与人类反馈（RLHF）。我们发现DPO本质上是一个由参数化策略类诱导的奖励函数统计估计问题。当真实奖励函数无法被策略类表示时，DPO出现误设，导致偏好顺序反转、策略奖励下降及对输入偏好分布高度敏感等失败模式。相反，我们分析了两阶段RLHF在参数化策略类下的局部行为，发现其等价于策略空间中的自然梯度步。基于这一精细几何刻画，我们提出辅助型DPO（AuxDPO），在损失函数中引入额外辅助变量，以有原则的方式逼近RLHF解，缓解DPO的误设问题。我们在教学性老虎机设置及大模型对齐任务上实证验证了AuxDPO的优越性。
原文摘要 · Abstract (English)
Direct alignment algorithms such as Direct Preference Optimization (DPO) fine-tune models based on preference data, using only supervised learning instead of two-stage reinforcement learning with human feedback (RLHF). We show that DPO encodes a statistical estimation problem over reward functions induced by a parametric policy class. When the true reward function that generates preferences cannot be realized via the policy class, DPO becomes misspecified, resulting in failure modes such as preference order reversal, worsening of policy reward, and high sensitivity to the input preference data distribution. On the other hand, we study the local behavior of two-stage RLHF for a parametric class and relate it to a natural gradient step in policy space. Our fine-grained geometric characterization allows us to propose AuxDPO, which introduces additional auxiliary variables in the DPO loss function to help move towards the RLHF solution in a principled manner and mitigate the misspecification in DPO. We empirically demonstrate the superior performance of AuxDPO on didactic bandit settings as well as LLM alignment tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。