改进2D-DPO模型，提升对标注噪声的鲁棒性。
Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
- 提出二维评分机制，区分响应中不同段落的偏好程度。
- 实验证明新方法在噪声环境下胜率提升12.3%。
- 适合需要高精度对齐的生成模型优化场景。
直接偏好优化（DPO）是一种强大的大语言模型对齐方法，相比基于人类反馈强化学习的方法更稳定高效。本文研究了开源偏好数据集上DPO的表现。现有DPO未引入细粒度评分，对响应各段同等对待，但实际人类偏好中即使优质回答也存在不被偏好的片段。为此，本文提出二维评分的DPO对齐范式——2D-DPO。通过比较胜率评估其优势。然而，该方法对标签/评分噪声仍不鲁棒。为此，本文提出在2D-DPO中融入段级评分噪声鲁棒性机制，提供理论支持与实证验证，并引入多种潜在噪声模型。
原文摘要 · Abstract (English)
Direct Preference Optimisation (DPO) has emerged as a powerful method for aligning Large Language Models (LLMs) with human preferences, offering a stable and efficient alternative to approaches that use Reinforcement learning via Human Feedback. In this work, we investigate the performance of DPO using open-source preference datasets. One of the major drawbacks of DPO is that it doesn't induce granular scoring and treats all the segments of the responses with equal propensity. However, this is not practically true for human preferences since even "good" responses have segments that may not be preferred by the annotator. To resolve this, a 2-dimensional scoring for DPO alignment called 2D-DPO was proposed. We explore the 2D-DPO alignment paradigm and the advantages it provides over the standard DPO by comparing their win rates. It is observed that these methods, even though effective, are not robust to label/score noise. To counter this, we propose an approach of incorporating segment-level score noise robustness to the 2D-DPO algorithm. Along with theoretical backing, we also provide empirical verification in favour of the algorithm and introduce other noise models that can be present.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。