为人类反馈强化学习的评分模型，实现个性化校准。
PEBS: Per-rater Empirical-Bayes Shrinkage for RLHF Reward-Model Calibration
- 基于经验贝叶斯方法，为每位评分者单独估计校准参数。
- 在PRISM数据集上降低8.58%的评分误差，PluriHarms上降9.66%。
- 无需重训练模型，可直接用于新评分的推理阶段。
人类反馈强化学习（RLHF）中的奖励模型通常将数千名标注者的偏好合并，并拟合一个全局仿射校准器，这会将具有系统性评分偏移和斜率差异的标注者合并为单一平均标注者，无法匹配任何个体。PEBS是一种针对每名标注者的经验贝叶斯收缩估计器：它在每个标注者独立保留的数据子集上拟合其局部仿射校准器，并通过Morris-James-Stein的经验贝叶斯收缩方法向群体均值靠拢，全程闭式求解，无需重新训练奖励模型。在PRISM数据集上，相比全局斜率基线，PEBS使用户内保留样本的RMSE降低8.58%；在PluriHarms有害性评分任务中（Qwen-2.5 base，同族模型），同样基线下提升9.66%。PEBS是RLHF奖励建模中一种无需重训练、仅估计推理时标注者级映射的闭式后处理校准方法。
原文摘要 · Abstract (English)
Reward models for Reinforcement Learning from Human Feedback (RLHF) pool preferences across thousands of annotators and fit one global affine calibrator, collapsing raters with systematically different rating-scale offsets and slopes into a single average-rater fit that does not match any individual annotator. PEBS is a per-rater empirical-Bayes shrinkage estimator: it fits per-rater affine calibrators on a held-out slice of each annotator's ratings and applies Morris-James-Stein empirical-Bayes shrinkage toward the population mean, in closed form and without retraining the reward model. On PRISM, PEBS reduces within-user held-out RMSE by 8.58% over the pooled population-slope baseline. The procedure replicates on PluriHarms harm ratings (Qwen-2.5 base, in-family) with a +9.66% RMSE reduction over the same population-slope baseline. PEBS is a closed-form post-hoc estimator for annotator-specific affine calibration in RLHF reward modeling; it leaves the reward base model unchanged and estimates only the rater-level map used at inference time for new ratings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。