解决偏好数据噪声问题，提升扩散模型对复杂人类偏好的对齐效果
Learning from Noisy Preferences: A Semi-Supervised Learning Approach to Direct Preference Optimization
- 将一致偏好对视为干净标签，冲突对作为带噪无标签数据进行半监督学习
- 在10个基准上超越SOTA，在无需额外标注或奖励模型下实现更优对齐
- 适合追求高精度人类偏好对齐的生成模型研究者和应用开发者
人类视觉偏好具有多维特性，涵盖美学、细节保真度与语义一致性。然而现有数据集仅提供单一整体标注，导致严重标签噪声：某些维度表现好但其他维度差的图像被简单标记为胜者或败者。我们理论证明，将多维偏好压缩为二值标签会产生冲突梯度信号，误导扩散直接偏好优化（DPO）。为此，我们提出半监督方法Semi-DPO，将一致对视为干净标签数据，冲突对作为噪声无标签数据。方法先在共识过滤的清洁子集上训练，再用该模型作为隐式分类器为噪声集生成伪标签，实现迭代优化。实验表明，Semi-DPO在10个基准上达到最先进性能，显著提升与复杂人类偏好的对齐能力，且训练中无需额外人工标注或显式奖励模型。代码与模型将开源于：https://github.com/L-CodingSpace/semi-dpo
原文摘要 · Abstract (English)
Human visual preferences are inherently multi-dimensional, encompassing aesthetics, detail fidelity, and semantic alignment. However, existing datasets provide only single, holistic annotations, resulting in severe label noise: images that excel in some dimensions but are deficient in others are simply marked as winner or loser. We theoretically demonstrate that compressing multi-dimensional preferences into binary labels generates conflicting gradient signals that misguide Diffusion Direct Preference Optimization (DPO). To address this, we propose Semi-DPO, a semi-supervised approach that treats consistent pairs as clean labeled data and conflicting ones as noisy unlabeled data. Our method starts by training on a consensus-filtered clean subset, then uses this model as an implicit classifier to generate pseudo-labels for the noisy set for iterative refinement. Experimental results demonstrate that Semi-DPO achieves state-of-the-art performance and significantly improves alignment with complex human preferences, without requiring additional human annotation or explicit reward models during training. We will release our code and models at: https://github.com/L-CodingSpace/semi-dpo
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。