改进DPO训练中偏好响应概率下降问题，提升对齐效果
Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization
- 在logit空间分析梯度更新机制，发现高曲率方向导致概率压缩
- 提出logits-SAM，仅扰动输出层，显著缓解概率压缩现象
- 兼容多种DPO变体，在多个模型和数据集上均有效
直接偏好优化（DPO）因其简洁性和训练稳定性，已成为对齐预训练大语言模型与人类偏好的主流方法。然而，近期发现的挤压效应（又称似然位移）会导致偏好响应的概率在训练过程中意外降低。为理解并缓解该现象，我们构建了一个在logit空间中的坐标级动态理论框架。分析表明，负梯度更新会沿高曲率方向迅速放大残差，这是挤压效应的根本原因；而尖锐性感知最小化（SAM）通过其曲率正则化作用可抑制该行为。基于此，我们研究了计算高效的logits-SAM，仅对输出层进行扰动，开销极小。在Pythia-2.8B、Mistral-7B和Gemma-2B-IT等多个模型及数据集上的广泛实验表明，logits-SAM能持续提升DPO的有效性，并可无缝集成到其他DPO变体中。代码已开源：https://github.com/RitianLuo/logits-sam-dpo。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) has emerged as a popular algorithm for aligning pretrained large language models with human preferences, owing to its simplicity and training stability. However, DPO suffers from the recently identified squeezing effect (also known as likelihood displacement), where the probability of preferred responses decreases unintentionally during training. To understand and mitigate this phenomenon, we develop a theoretical framework that models the coordinate-wise dynamics in logit space. Our analysis reveals that negative-gradient updates cause residuals to expand rapidly along high-curvature directions, which underlies the squeezing effect, whereas Sharpness-Aware Minimization (SAM) can suppress this behavior through its curvature-regularization effect. Building on this insight, we investigate logits-SAM, a computationally efficient variant that perturbs only the output layer with negligible overhead. Extensive experiments on Pythia-2.8B, Mistral-7B, and Gemma-2B-IT across multiple datasets and benchmarks demonstrate that logits-SAM consistently improves the effectiveness of DPO and integrates seamlessly with other DPO variants. Code is available at https://github.com/RitianLuo/logits-sam-dpo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。