提出新奖励模型，让AI更懂自己不确定在哪
Probabilistic Uncertain Reward Model
- 用概率分布替代传统评分，捕捉人类反馈的不确定性
- 在测试中胜率更高，且能坚持更长时间优化而不崩溃
- 适合需要可靠判断的高风险场景，如医疗或金融决策
基于人类反馈的强化学习（RLHF）是训练大语言模型的关键技术。然而，传统的基于布拉德利-特瑞模型（BTRM）的奖励模型在面对不一致标注或分布外样本时容易产生过度自信，导致奖励欺骗——策略模型盲目追求代理奖励，反而损害真实性能。本文提出概率不确定奖励模型（PURM），将布拉德利-特瑞模型推广为学习偏好数据中生成的奖励分布。我们理论上推导了PURM的损失函数，并引入一种新方法，通过分布重叠量化不确定性。实验表明，PURM在奖励准确性与不确定性估计上均优于现有方法，能支持更长时间的有效学习，在RLHF中达到更高的最大胜率。论文代码与数据已公开于 https://anonymous.4open.science/r/Probabilistic-Uncertain-Reward-Model/
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) is a critical technique for training large language models. However, conventional reward models based on the Bradley-Terry model (BTRM) often suffer from overconfidence when faced with inconsistent labels or out-of-distribution samples, leading to reward hacking, where the policy model blindly optimizes for proxy rewards while degrading true performance. This paper proposes the Probabilistic Uncertain Reward Model (PURM), which generalizes the Bradley-Terry model to learn the reward distributions that emerged from the preference data. We theoretically derive the loss function of PURM and introduce a novel method that uses the overlap between distributions to quantify uncertainty. Empirical results show that PURM outperforms existing methods with more accurate reward and sound uncertainty estimations, and sustains effective learning for more optimization steps and obtain higher maximum win rate in RLHF. The data and code of this paper are released at https://anonymous.4open.science/r/Probabilistic-Uncertain-Reward-Model/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。