提出一种新方法，让生成式奖励模型更快更准地评估多个回复优劣。
IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Models
- 通过分组对比选择与拒绝样本，实现点对点奖励评分
- 计算量从O(n²)降为O(n)，支持任意数量候选回复
- 性能媲美顶尖成对模型，适合需要高效反馈的RLHF场景
生成式奖励模型（GRMs）因可解释性及可通过强化学习（RL）优化，在奖励建模中表现优异。然而，主流成对GRMs在人类反馈强化学习（RLHF）中存在计算瓶颈：当对n个候选回复进行偏好校准或聚合时，需进行O(n²)次成对判断。为此，我们提出组间相对偏好建模（IRPM），基于强化学习扩展布拉德利-特瑞模型，从成对偏好数据训练点对点GRMs。IRPM通过对比被选与被拒样本组，生成可跨候选集比较的点对点奖励分数，使强化学习训练阶段的奖励评估复杂度降至O(n)，同时保持可解释性与可扩展性。实验表明，IRPM在RM-Bench、JudgeBench和RewardBench上达到当前最优点对点GRM性能，并接近领先成对GRM表现；后训练评估中亦取得显著提升，验证其有效性。
原文摘要 · Abstract (English)
Generative Reward Models (GRMs) have demonstrated strong performance in reward modeling, due to their interpretability and potential for refinement through reinforcement learning (RL). However, widely used pairwise GRMs create a computational bottleneck in reinforcement learning from human feedback (RLHF), when calibrating or aggregating preference signals over n candidates, often incurring O(n^2) pairwise judgments. To address this issue, we propose Intergroup Relative Preference Modeling (IRPM), an RL-based method that extends the Bradley--Terry preference-learning paradigm via intergroup comparisons to train pointwise GRMs from pairwise preference data. IRPM derives pointwise reward for each response by contrasting groups of chosen vs. rejected samples, enabling pointwise scores comparable across candidate sets and O(n) reward evaluation for a variable number of candidates during RL training, while preserving interpretability and scalability. Experiments show that IRPM achieves state-of-the-art performance among pointwise GRMs on RM-Bench, JudgeBench and RewardBench, and approaches the performance of leading pairwise GRMs. In addition, IRPM achieves substantial gains in post-training evaluations, demonstrating its effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。