用成对反馈训练点式奖励模型，提升大模型对齐效果
PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling
- 通过偏好感知机制，用成对数据实现点式训练
- 在RewardBench上提升8.7%，下游强化学习性能增13.6%
- 适合需要高效标注的RLHF研究者使用
奖励模型（RMs）是人类反馈强化学习（RLHF）的核心，为大语言模型（LLMs）与人类偏好对齐提供关键监督信号。生成式奖励模型（GRMs）比传统标量奖励模型更具可解释性，但存在关键权衡：成对方法存在训练-推理不一致问题，点式方法则需昂贵的绝对评分标注。为此，我们提出偏好感知任务自适应奖励模型（PaTaRM）。不同于以往方法，PaTaRM通过新颖的偏好感知奖励（PAR）机制，利用现有成对数据实现稳健的点式训练，无需显式评分标签。同时引入任务自适应评分体系，动态生成实例级评估标准。大量实验表明，PaTaRM在Qwen3-8B/14B模型上于RewardBench和RMBench平均提升8.7%；更重要的是，在IFEval和InFoBench上使下游RLHF性能平均相对提升13.6%，验证了其在策略对齐中的有效性。代码已公开于https://github.com/JaneEyre0530/PaTaRM。
原文摘要 · Abstract (English)
Reward models (RMs) are central to reinforcement learning from human feedback (RLHF), providing the critical supervision signals that align large language models (LLMs) with human preferences. Generative reward models (GRMs) provide greater interpretability than traditional scalar RMs, but they come with a critical trade-off: pairwise methods are hindered by a training-inference mismatch, while pointwise methods require expensive absolute annotations. To bridge this gap, we propose the Preference-aware Task-adaptive Reward Model (PaTaRM). Unlike prior approaches, PaTaRM enables robust pointwise training using readily available pairwise data via a novel Preference-Aware Reward (PAR) mechanism, eliminating the need for explicit rating labels. Furthermore, it incorporates a Task-Adaptive Rubric system that dynamically generates instance-specific criteria for precise evaluation. Extensive experiments demonstrate that PATRM achieves a 8.7% average improvement on RewardBench and RMBench across Qwen3-8B/14B models. Crucially, it boosts downstream RLHF performance by an average relative improvement of 13.6% across IFEval and InFoBench, validating its effectiveness for policy alignment. Our code is available at https://github.com/JaneEyre0530/PaTaRM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。