双向评估提升大模型推理评分精度,仅增0.3%参数和5%推理延迟。
The Bidirectional Process Reward Model
- 引入左右双向评估流,通过提示反转实现并行推理
- 在54种配置中平均提升10.6%,步级错误检测提升37.7%
- 轻量设计适合部署,适用于各类大模型与推理任务
过程奖励模型(PRM)通过为解题轨迹中的中间推理步骤分配细粒度评分,成为提升大语言模型(LLM)推理质量的有力方法。然而,现有大多数PRM采用单向从左到右(L2R)评估方式,限制了全局上下文的利用。针对此问题,我们提出一种新型双向评估范式——双向过程奖励模型(BiPRM)。BiPRM在传统L2R流程外,引入并行的从右到左(R2L)评估流,通过提示反转实现。随后,通过门控机制自适应融合双流奖励分数,实现更全面的质量评估。值得注意的是,相比原PRM,BiPRM仅增加0.3%参数(用于门控模块),双流并行执行带来的推理延迟增加仅为5%。在多个基准测试、不同LLM主干、不同PRM目标和采样策略下的广泛实证评估表明,BiPRM始终优于单向基线,在54种解题级别配置中平均相对提升10.6%,在12种步骤级错误检测场景中提升达37.7%。结果表明,BiPRM具有显著有效性、鲁棒性与通用性,为基于过程的奖励建模提供了新方向。
原文摘要 · Abstract (English)
Process Reward Models (PRMs), which assign fine-grained scores to intermediate reasoning steps within a solution trajectory, have emerged as a promising approach to enhance the reasoning quality of Large Language Models (LLMs). However, most existing PRMs rely on a unidirectional left-to-right (L2R) evaluation scheme, which restricts their utilization of global context. In light of this challenge, we propose a novel bidirectional evaluation paradigm, named Bidirectional Process Reward Model (BiPRM). BiPRM incorporates a parallel right-to-left (R2L) evaluation stream, implemented via prompt reversal, alongside the conventional L2R flow. Then a gating mechanism is introduced to adaptively fuse the reward scores from both streams to yield a holistic quality assessment. Remarkably, compared to the original PRM, BiPRM introduces only a 0.3% parameter increase for the gating module, and the parallel execution of two streams incurs merely 5% inference time latency. Our extensive empirical evaluations spanning diverse benchmarks, LLM backbones, PRM objectives and sampling policies demonstrate that BiPRM consistently surpasses unidirectional baselines, achieving an average relative gain of 10.6% over 54 solution-level configurations and 37.7% in 12 step-level error detection scenarios. Generally, our results highlight the effectiveness, robustness and general applicability of BiPRM, offering a promising new direction for process-based reward modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。