提出首个可证明鲁棒对齐的损失函数，能自动识别并剔除人类反馈中的噪声标签。
Scalable Valuation of Human Feedback through Provably Robust Model Alignment
- 设计霍尔德DPO损失，具备抗噪声能力，确保在严重标签错误下仍能收敛到正确参数。
- 在可控数据集上准确检测出约15%的错误标签，且移除后模型性能显著提升。
- 无需人工校验即可大规模评估人类反馈质量，适合需要高可靠标注的场景。
尽管语言模型与人类偏好对齐至关重要，但众包人类反馈常含噪声——例如偏好较差回复——构成根本性挑战。真正鲁棒的对齐目标应在严重标签噪声下仍产生相同模型参数，此性质称为红下降（redescending）。我们证明现有对齐方法均不满足该性质。为此，提出首个具有理论保障红下降特性的霍尔德DPO（Hölder-DPO）损失，可从噪声反馈中估计出清洁数据分布。对齐模型能估计清洁数据似然，提供理论支持的数据集估值指标，精确识别错误标签的位置与比例。该指标无需梯度计算，实现无需人工校验、低成本的可扩展人类反馈估值。霍尔德DPO在多个基准上达到当前最优鲁棒对齐表现，并在控制数据集中准确检测出约15%的误标。应用于Anthropic HH-RLHF数据集时，发现显著噪声水平，去除这些误标后，各方法对齐性能均有明显提升。代码已开源。
原文摘要 · Abstract (English)
Despite the importance of aligning language models with human preferences, crowd-sourced human feedback is often noisy -- for example, preferring less desirable responses -- posing a fundamental challenge to alignment. A truly robust alignment objective should yield identical model parameters even under severe label noise, a property known as redescending. We prove that no existing alignment methods satisfy this property. To address this, we propose Hölder-DPO, the first principled alignment loss with a provable redescending property, enabling estimation of the clean data distribution from noisy feedback. The aligned model estimates the likelihood of clean data, providing a theoretically grounded metric for dataset valuation that identifies the location and fraction of mislabels. This metric is gradient-free, enabling scalable and automated human feedback valuation without costly manual verification or clean validation dataset. Hölder-DPO achieves state-of-the-art robust alignment performance while accurately detecting mislabels in controlled datasets. Finally, applied to Anthropic HH-RLHF dataset, it reveals substantial noise levels and removing these mislabels significantly improves alignment performance across methods. The code is available at https://github.com/ma921/HolderDPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。