PRM模型常因训练数据偏差误奖错误推理步骤，新方法PRISM通过对比学习减少误判。
The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning
- 用对比学习和时序负样本替代传统标签拟合，提升步骤评估可靠性
- 在PRMBench上将误报率降低22%，宏F1指标显著优于现有模型
- 适合需要精准推理监督的场景，如智能体决策、生成优化与策略训练
过程奖励模型（PRMs）通过提供分步反馈改善推理任务中的信用分配。然而我们发现，由于步骤级训练数据严重失衡，标准交叉熵训练会放大这一偏差，导致模型过度奖励看似合理但错误的步骤，产生高误报率。这些误报具有不对称的下游影响：误报会主动引导Best-of-N选择、引导解码和策略优化走向错误推理，而漏报仅减缓探索速度。这表明PRM训练应从逐点标签拟合转向可靠的相对比较。为此，我们提出PRISM（精度排序改进步骤建模），一种政策感知的PRM训练框架，通过对比学习和时序前瞻生成的硬负样本进行训练，无需新增人工标注。进一步采用难度感知课程优化对比边界。在PRMBench和ProcessBench上，PRISM显著降低误报率（PRMBench上降低22%），并提升宏观F1分数。应用于策略优化与搜索任务（包括引导解码和Best-of-N选择）时，准确率持续提升（引导解码最高提升22%，Best-of-N提升33%），且更具鲁棒性。更广泛而言，可信的过程监督不仅在于给予高奖励，更在于为正确的推理原因赋予奖励。
原文摘要 · Abstract (English)
Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback. However, we identify a hidden bias in PRMs caused by severe imbalance in step-level training data. Standard cross-entropy training amplifies this bias, causing PRMs to overcredit plausible but incorrect steps and produce high false-positive rates. We show that these false positives have an asymmetric downstream effect: false negatives mainly slow exploration, whereas false positives actively steer Best-of-N selection, guided decoding, and policy optimization toward flawed reasoning. This suggests that PRM training should shift from pointwise label fitting to reliable relative comparisons. To address this, we propose PRISM (Precision Ranking for Improved Step Modeling), a policy-aware PRM training framework that learns from contrastive step-level comparisons and hard negatives generated by a temporal lookahead strategy, requiring no new human labels. We further use a difficulty-aware curriculum to optimize the contrastive step margin. Across PRMBench and ProcessBench, PRISM substantially reduces false positives (22% on PRMBench) and improves macro F1 over strong discriminative PRMs. When applied to policy optimization and search tasks, including guided decoding and Best-of-N selection, it consistently improves accuracy (up to 22% for guided decoding and 33% for Best-of-N) and robustness. More broadly, trustworthy process supervision is not just about assigning high rewards, but about rewarding the right reasoning for the right reasons.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。