高质数据能增强策略优化梯度信号，提升直接偏好优化效果
Understanding the Impact of Sampling Quality in Direct Preference Optimization
- 通过简化模型避免似然位移，更清晰分析数据质量对更新的影响
- 高质量响应越频繁，梯度信号越强，优化路径越高效
- 为在线DPO框架提供理论依据，适合做对齐训练的工程师参考
我们研究了如何利用更高品质的数据来提升直接偏好优化（DPO）的性能，旨在理解其对DPO训练动态的影响。分析表明，DPO的解空间和收敛行为依赖于数据生成分布的支持范围与质量。我们首先分析了数据与参考策略如何影响梯度下降中的策略更新，并揭示了一种称为似然位移的实际现象会干扰预期动态。随后，我们设计了一个简化但结构良好的对齐模型作为代理，保留了强化学习人类反馈（RLHF）的多数优点，同时避免了似然位移。基于该模型，我们获得了定量结果：高频出现的高质量响应能显著放大梯度信号，改善优化景观，从而实现更有效的策略学习。理论发现经实证实验验证，为实践中在线DPO框架提供了原则性支持。
原文摘要 · Abstract (English)
We study how data of higher quality can be leveraged to improve performance in Direct Preference Optimization (DPO), aiming to understand its impact on DPO training dynamics. Our analyses show that both the solution space and the convergence behavior of DPO depend on the support and quality of the data-generating distribution. We first analyze how data and reference policy influence policy updates during gradient descent, and how a practical phenomenon known as likelihood displacement can interfere with the desired dynamics. We then design a simplified yet well-structured alignment model as a proxy that preserves most of the beneficial properties of RLHF while avoiding likelihood displacement. Based on this model, we develop quantitative results showing how more frequent high-quality responses amplify the gradient signal and improve the optimization landscape, leading to more effective policy learning. Our theoretical findings are supported by empirical experiments and provide a principled justification for the online DPO framework in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。