不同采样策略显著影响DPO收敛速度，新方法实现二次收敛。
The Crucial Role of Samplers in Online Direct Preference Optimization
- 对比均匀采样与在线采样，发现后者收敛更快。
- 新方法在Safe-RLHF数据集上比原版DPO提升超7.4%。
- 适合关注模型对齐优化理论与实践的研究者。
直接偏好优化（DPO）已成为一种稳定、可扩展且高效的语言模型对齐方案。尽管其在实践中取得成功，但其优化特性，尤其是采样策略对其收敛速度的影响仍缺乏深入研究。本文在精确梯度设定下，对不同采样策略的DPO收敛速率进行了严格分析，揭示了一个令人惊讶的差异：均匀采样实现线性收敛，而我们提出的在线采样则达到二次收敛。进一步地，通过引入后验分布和逻辑值混合，将该采样器适配到实际场景，结果表明其在Safe-RLHF数据集上相较原始DPO性能提升超过7.4%。研究不仅深化了对DPO的理论理解，也为后续算法设计提供了新方向。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) has emerged as a stable, scalable, and efficient solution for language model alignment. Despite its empirical success, the optimization properties, particularly the impact of samplers on its convergence rates, remain under-explored. In this paper, we provide a rigorous analysis of DPO's convergence rates with different sampling strategies under the exact gradient setting, revealing a surprising separation: uniform sampling achieves $\textbf{linear}$ convergence, while our proposed online sampler achieves $\textbf{quadratic}$ convergence. We further adapt the sampler to practical settings by incorporating posterior distributions and logit mixing, demonstrating improvements over previous methods. For example, it outperforms vanilla DPO by over $7.4$% on Safe-RLHF dataset. Our results not only offer insights into the theoretical understanding of DPO but also pave the way for further algorithm designs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。