通过惩罚不确定偏好，提升大模型对人类偏好的对齐效果。
Uncertainty-Penalized Direct Preference Optimization
- 引入不确定性惩罚机制，修正DPO损失以应对模糊或错误标注的偏好对。
- 在Anthropic-HH数据集上，相比原版DPO性能更优，尤其在高不确定性样本上生成更合理内容。
- 适合关注模型对齐鲁棒性、避免奖励黑客的研究者使用。
将大型语言模型（LLMs）与人类在内容、风格和呈现上的偏好对齐极具挑战性，部分原因在于偏好具有多样性、上下文依赖性且常具内在模糊性。尽管表现良好，基于人类反馈的强化学习（RLHF）和直接偏好优化（DPO）易出现代理奖励过优化问题。对DPO损失的分析揭示了对误标注或模糊偏好对进行正则化的必要性，以防止奖励欺骗。本文受离线强化学习启发，提出一种悲观化DPO框架，引入偏好不确定性惩罚机制，该机制通过降低不确定样本的损失梯度来修正损失。实验在GPT2 Medium模型上，使用模型集成估计不确定性，在Anthropic-HH数据集上验证，结果表明该方法优于原始DPO，且在高不确定性选择/拒绝响应的提示下生成效果更佳。
原文摘要 · Abstract (English)
Aligning Large Language Models (LLMs) to human preferences in content, style, and presentation is challenging, in part because preferences are varied, context-dependent, and sometimes inherently ambiguous. While successful, Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are prone to the issue of proxy reward overoptimization. Analysis of the DPO loss reveals a critical need for regularization for mislabeled or ambiguous preference pairs to avoid reward hacking. In this work, we develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes, inspired by offline reinforcement learning. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. Evaluation of the methods is performed with GPT2 Medium on the Anthropic-HH dataset using a model ensemble to obtain uncertainty estimates, and shows improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。