解决大模型对齐中优选响应概率下降问题，提升生成质量。
DPO-Shift: Shifting the Distribution of Direct Preference Optimization
- 通过可控分布调整优化优选响应概率
- 在保持奖励差距的同时提升优选概率
- 适合关注模型对齐与生成稳定性的研究者
直接偏好优化（DPO）及其变体已成为对齐语言模型与人类偏好的主流方法。这类方法旨在让模型更好地区分优选与次优响应。然而，已有研究表明，训练过程中优选响应的概率常出现下降，即存在似然位移现象。本文提出 DPO-Shift，可控制地调整优选概率的分布。理论分析与实验验证均表明，该方法在提升优选概率的同时会牺牲奖励差距，存在根本性权衡。此外，在 MT-Bench 和自定义胜率测试中，DPO-Shift 均优于原始 DPO。我们相信，该工作为缓解 DPO 的似然位移问题提供了简单且理论严谨的解决方案。代码已开源。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) and its variants have become increasingly popular for aligning language models with human preferences. These methods aim to teach models to better distinguish between chosen (or preferred) and rejected (or dispreferred) responses. However, prior research has identified that the probability of chosen responses often decreases during training, and this phenomenon is known as likelihood displacement. To tackle this challenge, in this work we introduce DPO-Shift to controllably shift the distribution of the chosen probability. Then, we show that DPO-Shift exhibits a fundamental trade-off between improving the chosen probability and sacrificing the reward margin, as supported by both theoretical analysis and experimental validation. Furthermore, we demonstrate the superiority of DPO-Shift over DPO on downstream tasks such as MT-Bench and a designed win rate experiment. We believe this study shows that the likelihood displacement issue of DPO can be effectively mitigated with a simple, theoretically grounded solution. Our code is available at https://github.com/Meaquadddd/DPO-Shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。