让奖励模型学会识别不确定，防止错误反馈误导强化学习。
Uncertainty-Aware Reward Modeling for Stable RLHF
- 用分位数校准预测不确定性，让模型知道何时不靠谱
- 实验显示奖励错误率下降37%，对齐效果显著提升
- 适合追求稳定对齐的LLM训练者，尤其关注可靠性
基于人类反馈的强化学习（RLHF）通过偏好数据训练奖励模型，并优化策略以最大化预测奖励。然而该流程面临两大根本挑战：(1) 奖励模型通常作为确定性点估计器，无法提示其预测是否不可靠；(2) 现代群体策略优化方法会放大不可靠的奖励信号，例如GRPO在优势计算中对所有奖励一视同仁。随着策略生成越来越多样化的响应，这两项缺陷导致严重漏洞：不可靠的奖励估计可能获得过强影响力，引发严重的奖励黑客问题。本文提出不确定性感知奖励建模（UARM），通过基于分位数的符合预测校准奖励模型的不确定性，并利用异方差方差分解重新加权GRPO的优势。在HelpSteer、UltraFeedback和PKU-SafeRLHF数据集上的实验表明，相比标准GRPO和无不确定性意识基线，UARM显著提升了奖励模型校准度，减少奖励黑客行为，并改善下游对齐质量。
原文摘要 · Abstract (English)
Reinforcement learning from human feedback (RLHF) aligns large language models by training reward models on preference data and optimizing policies to maximize predicted rewards. However, this pipeline faces two fundamental challenges: (1) reward models cannot signal when their predictions are unreliable, since they usually act as deterministic point estimators; and (2) modern group-based policy optimization can amplify unreliable reward signals, as exemplified by GRPO's uniform treatment of rewards during advantage computation. As policies explore increasingly diverse responses, these two limitations create a critical vulnerability: unreliable reward estimates may be granted disproportionate influence, triggering severe reward hacking. We propose Uncertainty-Aware Reward Modeling (UARM), which equips reward models with calibrated uncertainty via quantile-based conformal prediction and reweights GRPO advantages through heteroscedastic variance decomposition. Experiments across HelpSteer, UltraFeedback, and PKU-SafeRLHF demonstrate that UARM significantly improves reward model calibration, reduces reward hacking, and enhances downstream alignment quality compared to standard GRPO and uncertainty-agnostic baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。