双向奖励信号提升大模型推理步骤评估精度
Better Process Supervision with Bi-directional Rewarding Signals
- 引入双向奖励机制，同时评估已走步骤与未来成功概率
- 在Gaokao2023上比PRM提升3.1%，在MATH-500上比PRM高3.8%
- 适合需要精细过程监督的数学推理与搜索策略场景
过程监督对复杂大语言模型推理和测试时搜索至关重要。现有方法（如过程奖励模型）仅基于当前步骤奖励，具有单向性且无法建模距目标的距离。受A*算法启发，我们提出BiRM模型，不仅评估历史步骤正确性，还预测未来成功率。在数学推理任务上，BiRM在Best-of-N采样下使Gaokao2023准确率提升3.1%；在搜索策略中，相较ORM提升5.0%，较PRM提升3.8%，显著改善推理指导能力。
原文摘要 · Abstract (English)
Process supervision, i.e., evaluating each step, is critical for complex large language model (LLM) reasoning and test-time searching with increased inference compute. Existing approaches, represented by process reward models (PRMs), primarily focus on rewarding signals up to the current step, exhibiting a one-directional nature and lacking a mechanism to model the distance to the final target. To address this problem, we draw inspiration from the A* algorithm, which states that an effective supervisory signal should simultaneously consider the incurred cost and the estimated cost for reaching the target. Building on this key insight, we introduce BiRM, a novel process supervision model that not only evaluates the correctness of previous steps but also models the probability of future success. We conduct extensive experiments on mathematical reasoning tasks and demonstrate that BiRM provides more precise evaluations of LLM reasoning steps, achieving an improvement of 3.1% on Gaokao2023 over PRM under the Best-of-N sampling method. Besides, in search-based strategies, BiRM provides more comprehensive guidance and outperforms ORM by 5.0% and PRM by 3.8% respectively on MATH-500.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。