提出更稳定的语言模型对齐损失，解决DPO易崩溃问题
A Principled Loss Function for Direct Language Model Alignment
- 从强化学习最优性条件推导新损失函数
- 控制logits差值在有限目标值，避免无限放大
- 训练更稳定，适合追求可靠对齐的开发者
大语言模型与人类偏好对齐通常通过人类反馈强化学习（RLHF）实现。直接偏好优化（DPO）简化了这一范式，建立了最优策略与奖励函数之间的直接映射，无需显式奖励模型。然而，我们认为DPO损失函数在理论上与其推导不一致，因其促进对logits差值的无限最大化，可能导致训练不稳定和奖励滥用。本文提出一种新损失函数，直接源于RLHF的最优性条件。该损失针对由底层奖励决定的特定、有限的logits差值目标，而非其最大化。我们提供了理论分析，包括基于梯度的比较，证明本方法可避免当非优选响应概率趋近零时出现的大梯度问题。这种内在稳定性防止了奖励滥用，实现了更有效的对齐。通过微调Qwen2.5-7B模型验证，相比标准DPO基线显著提升胜率，并达到与更大模型Llama-3.1-8B相当的性能。
原文摘要 · Abstract (English)
The alignment of large language models (LLMs) with human preferences is commonly achieved through Reinforcement Learning from Human Feedback (RLHF). Direct Preference Optimization (DPO) simplified this paradigm by establishing a direct mapping between the optimal policy and a reward function, eliminating the need for an explicit reward model. However, we argue that the DPO loss function is theoretically misaligned with its own derivation, as it promotes the indefinite maximization of a logits difference, which can lead to training instability and reward hacking. In this paper, we propose a novel loss function derived directly from the RLHF optimality condition. Our proposed loss targets a specific, finite value for the logits difference, which is dictated by the underlying reward, rather than its maximization. We provide a theoretical analysis, including a gradient-based comparison, to demonstrate that our method avoids the large gradients that plague DPO when the probability of dispreferred responses approaches zero. This inherent stability prevents reward hacking and leads to more effective alignment. We validate our approach by fine-tuning a Qwen2.5-7B model, showing significant win-rate improvements over a standard DPO baseline and achieving competitive performance against larger models like Llama-3.1-8B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。