系统梳理无需强化学习的对齐方法DPO，涵盖数据、理论与应用。
A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
- 按研究问题分类总结DPO的理论与变体
- 归纳现有偏好数据集及实际应用场景
- 适合关注大模型对齐的科研人员参考
随着大语言模型（LLMs）的快速发展，将策略模型与人类偏好对齐变得愈发关键。直接偏好优化（DPO）作为一种无需强化学习的对齐方法，成为强化学习从人类反馈（RLHF）的有力替代方案。尽管DPO在多个方面取得进展并存在固有局限，但当前文献中尚缺乏对其全面深入的综述。本文系统回顾了DPO面临的挑战与机遇，涵盖理论分析、各类变体、相关偏好数据集及实际应用。我们根据核心研究问题对近期DPO研究进行分类，以全面呈现其发展现状。此外，本文还提出若干未来研究方向，为模型对齐领域提供前瞻性洞见。相关论文合集可访问 https://github.com/Mr-Loevan/DPO-Survey。
原文摘要 · Abstract (English)
With the rapid advancement of large language models (LLMs), aligning policy models with human preferences has become increasingly critical. Direct Preference Optimization (DPO) has emerged as a promising approach for alignment, acting as an RL-free alternative to Reinforcement Learning from Human Feedback (RLHF). Despite DPO's various advancements and inherent limitations, an in-depth review of these aspects is currently lacking in the literature. In this work, we present a comprehensive review of the challenges and opportunities in DPO, covering theoretical analyses, variants, relevant preference datasets, and applications. Specifically, we categorize recent studies on DPO based on key research questions to provide a thorough understanding of DPO's current landscape. Additionally, we propose several future research directions to offer insights on model alignment for the research community. An updated collection of relevant papers can be found on https://github.com/Mr-Loevan/DPO-Survey.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。