通过分步错误识别提升数学推理奖励模型性能，更准且更省数据。
Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision
- 分层设计：先分类每步的错误类型，再综合判断正确性
- 在PRMBench上达67.7分新高，用3倍少数据超越前人
- 适合做数学推理生成的强化学习奖励机制优化
大型语言模型在多跳推理任务中易产生幻觉。传统结果型奖励模型仅评估最终答案，而过程奖励模型（PRM）可对每一步进行评分以引导生成。本文提出PathFinder-PRM，一种分层、错误感知的判别式PRM：先在每一步识别数学错误与一致性错误，再融合细粒度信号估计步骤正确性。为训练该模型，我们构建了一个40万样本数据集，基于人类标注的PRM800K语料与RLHFlow Mistral轨迹，加入三维步骤级标签。在PRMBench上，PathFinder-PRM达到67.7的新纪录，优于此前最佳的65.5，且仅使用三倍少的数据量。应用于奖励引导贪婪搜索时，其prm@8达48.3，较最强基线提升1.5分。结果表明，解耦错误检测与奖励估计不仅能显著提升细粒度错误识别能力，还能在端到端数学推理中实现更高数据效率。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are prone to hallucination, especially during multi-hop and reasoning-intensive tasks such as mathematical problem solving. While Outcome Reward Models verify only final answers, Process Reward Models (PRMs) score each intermediate step to steer generation toward coherent solutions. We introduce PathFinder-PRM, a novel hierarchical, error-aware discriminative PRM that first classifies math and consistency errors at each step, then combines these fine-grained signals to estimate step correctness. To train PathFinder-PRM, we construct a 400K-sample dataset by enriching the human-annotated PRM800K corpus and RLHFlow Mistral traces with three-dimensional step-level labels. On PRMBench, PathFinder-PRM achieves a new state-of-the-art PRMScore of 67.7, outperforming the prior best (65.5) while using 3 times less data. When applied to reward guided greedy search, our model yields prm@8 48.3, a +1.5 point gain over the strongest baseline. These results demonstrate that decoupled error detection and reward estimation not only boost fine-grained error detection but also substantially improve end-to-end, reward-guided mathematical reasoning with greater data efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。