arXiv:2510.00492cs.AI2025-10被引 8

对比四种奖励模型,发现生成式结果验证更适配多领域部署。

Rethinking Reward Models for Multi-Domain Test-Time Scaling

  • 比较四种奖励模型在14个领域的表现,统一评估标准。
  • 生成式结果模型(gORM)在所有领域均显著优于其他模型。
  • 细粒度推理评分易受噪声影响,长推理链效果更差。

大语言模型在测试阶段扩展时的可靠性常通过外部验证器或奖励模型来评估,以区分正确推理与错误逻辑。现有研究关注仅评价最终答案的结果奖励模型(ORM)和评估中间推理步骤的过程奖励模型(PRM)。尽管PRM因更细粒度的监督常被认为更优,但多数证据来自数学相关场景,其跨领域优势尚不明确。本文首次在14个多样化领域中统一评估四类奖励模型:判别式与生成式版本的ORM和PRM(dORM、dPRM、gORM、gPRM)。结果表明:(i) dORM性能与dPRM相当;(ii) gPRM表现不佳;(iii) gORM整体最稳健,在所有测试领域均带来显著且一致的提升。我们归因于gPRM的分步评分机制继承了基于LLM自动标注的标签噪声,导致长推理轨迹(包括自修正推理)难以准确评估。理论分析与实证观察均显示,分步聚合会随推理长度增加而放大误差。该发现挑战了‘细粒度监督更优’的普遍假设,支持采用生成式结果验证用于多领域部署。代码已公开。

原文摘要 · Abstract (English)

The reliability of large language models (LLMs) during test-time scaling is often assessed with \emph{external verifiers} or \emph{reward models} that distinguish correct reasoning from flawed logic. Prior work has studied both outcome reward models (ORMs), which assess only the final answer, and process reward models (PRMs), which score intermediate reasoning steps. Although PRMs are often viewed as advantageous due to their finer-grained supervision, much of the supporting evidence comes from math-adjacent settings, and their relative benefits across broader domains remain unclear. We present the first unified evaluation of four reward model variants, discriminative ORM and PRM (dORM, dPRM) and generative ORM and PRM (gORM, gPRM), across 14 diverse domains. Contrary to conventional wisdom, we find that (i) dORM performs on par with dPRM, (ii) gPRM is not competitive, and (iii) overall, gORM is the most robust, yielding significant and consistent gains across every tested domain. We attribute the worse performance of gPRM to the stepwise scoring process, which inherits label noise from LLM-based automatic labeling, leading to difficulties in evaluating long reasoning trajectories, including those involving self-correcting reasoning. Both our theoretical analysis and empirical observations indicate that stepwise aggregation compounds errors as reasoning length increases. These findings challenge the common assumption that fine-grained supervision is always better and support generative outcome verification for multi-domain deployment. Our \href{https://github.com/db-Lee/Multi-RM}{\underline{code}} is publicly available to facilitate future research in multi-domain settings.

奖励模型多领域推理验证LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。