让奖励模型学会预测每一步的未来价值，提升可解释性与效率
Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling
- 通过双重正则化使奖励模型输出符合条件期望
- 中间词对判别准确率从50%提升至88.9%，最终词准确率不变
- 无需修改架构即可实现解释性增强与训练加速
强化学习中人类反馈（RLHF）的奖励模型通常仅对响应的最后一个词进行评分，这忽略了中间位置的丰富信号，导致令牌级输出无意义。我们提出时序一致奖励建模（TCRM），通过在标准Bradley-Terry损失上添加两项正则化，使模型在任意时刻的输出等于给定当前响应的未来奖励条件期望，其最小值理论上即为条件期望。正则项对应蒙特卡洛和时序差分价值学习目标，直接建立与强化学习价值函数的联系。TCRM无需改变网络结构、数据或推理方式，却能实现三大优势：中间词对判别准确率由50%提升至88.9%，最终词准确率保持不变；在ProcessBench上以仅基于结果数据训练的方式达到44.9%平均F1，性能领先；在PPO中统一建模奖励与价值，峰值显存降低27%，单步耗时减少19%，且大模型质量不变。
原文摘要 · Abstract (English)
Reward models in RLHF are trained to score only the final token of a response - a choice that discards rich signal from every intermediate position and produces models whose token-level outputs are noise. We argue this is a missed opportunity: a well-trained reward model's output at any token should represent the conditional expectation of the final reward given the response so far. We introduce Temporally Coherent Reward Modeling (TCRM), which induces this property via two regularization terms on top of the standard Bradley-Terry loss, with minimizers provably equal to conditional expectations. The regularizers correspond to Monte Carlo and TD value-learning objectives, establishing a direct connection to RL value functions. TCRM requires zero changes to architecture, data, or inference, yet unlocks three capabilities from one principle: interpretable token-level reward trajectories (middle-token pairwise accuracy improved from 50% to 88.9%, final-token accuracy preserved); state-of-the-art PRM performance on ProcessBench (44.9% average F1) among models trained only on outcome data; and unified reward/value modeling in PPO, reducing peak GPU memory by 27% and step time by 19% with matching LLM quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。