提出新方法，让大模型在含噪偏好数据下仍能准确对齐人类意图。
Unbiased Alignment for Large Language Models with Noisy Preferences
- 设计无偏奖励模型和无偏直接偏好优化损失，数学修正偏好噪声影响。
- 在多个数据集上超越现有方法，噪声环境下性能更稳定。
- 适合处理真实世界含噪反馈数据的研究者或工业落地场景。
大语言模型与人类偏好的对齐通常通过基于人类反馈的强化学习或直接偏好优化实现，但这些方法易受真实世界偏好数据中普遍存在的噪声影响。为解决这一关键问题，本文提出无偏对齐的理论框架，引入无偏奖励模型（URM）损失和无偏直接偏好优化（UDPO）损失。通过数学方式纠正偏好噪声引起的偏差，新目标可直接从含噪数据中进行无偏训练，无需干净的真值监督。我们提供了严格的理论分析，证明所提方法具有抗噪声性、参数向下兼容性和分类校准性。在多个数据集上的全面实验表明，该方法优于当前最优基线。代码已开源：https://github.com/cswjl/unbiased-alignment。
原文摘要 · Abstract (English)
The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: https://github.com/cswjl/unbiased-alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。