动态调整数据权重，让大模型学得更准更快
Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
- 根据数据质量和模型学习进度自适应重加权偏好样本
- 在Arena-Hard上超越Claude 3 Opus 6.7分，数学与多模态任务全面领先
- 适合追求高效微调和强泛化能力的LLM研究者与开发者
直接偏好优化（DPO）因其简洁高效成为人类反馈强化学习（RLHF）的核心方法。然而，现有DPO方法通常对所有偏好对等同处理，忽视了数据质量与学习难度的显著差异，导致数据利用效率低、性能不佳。为此，我们提出统一动态偏好优化框架Uni-DPO，同时考虑（a）偏好对的固有质量与（b）模型训练过程中的动态表现。通过联合调整样本权重，Uni-DPO实现更高效的数据利用并取得更优性能。大量实验表明其有效性和通用性：在文本任务中，使用Uni-DPO微调的Gemma-2-9B-IT在Arena-Hard上比领先的Claude 3 Opus高出6.7分；在数学与多模态任务中，其在所有基准测试中均持续优于基线方法，充分验证了其有效性与鲁棒性。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) has emerged as a cornerstone of reinforcement learning from human feedback (RLHF) due to its simplicity and efficiency. However, existing DPO-based methods typically treat all preference pairs equally, overlooking substantial variations in data quality and learning difficulty, which leads to inefficient data utilization and suboptimal performance. To address this limitation, we propose Uni-DPO, a unified dynamic preference optimization framework that jointly considers (a) the inherent quality of preference pairs and (b) the model's evolving performance during training. By adaptively reweighting samples based on both factors, Uni-DPO enables more effective use of preference data and achieves superior performance. Extensive experiments across models and benchmarks demonstrate the effectiveness and generalization of Uni-DPO. On textual tasks, Gemma-2-9B-IT fine-tuned with Uni-DPO surpasses the leading LLM, Claude 3 Opus, by 6.7 points on Arena-Hard. On mathematical and multimodal tasks, Uni-DPO consistently outperforms baseline methods across all benchmarks, providing strong empirical evidence of its effectiveness and robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。