arXiv:2606.27739cs.LG2026-06

通过可学习的信用分配,让大模型自动生成推理过程评分。

The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment

论文配图:The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment
图 1 · 摘自论文原文
  • 设计可学习的信用分配机制,按最弱环节原则分配结果责任。
  • 在多个任务上超越现有方法,显著提升错误推理步骤识别率。
  • 适合希望低成本优化大模型推理能力的研究者使用。

过程奖励模型(PRMs)通过细粒度反馈增强大语言模型(LLMs)的推理能力,但训练通常需要昂贵的步骤级标注。结果监督型PRMs通过仅依赖最终答案正确性实现可扩展训练,却面临根本性的信用分配挑战——如何将结果归因于具体的推理步骤。现有方法采用均匀或因果分配,均无法准确锚定信用于步骤正确性,从而阻碍过程错误识别。本文提出基于可学习信用分配(LCA)的结果监督型PRM框架,遵循‘最弱环节原则’:推理链的强度取决于其最弱步骤。为解决信用分配与奖励建模间的相互依赖,我们将问题形式化为多实例学习(MIL),引入专为高依赖性和冗余性设计的Softmax加权求和(SWS)池化。在温和假设下证明了算法的贝叶斯一致性。大量实验表明,LCA在多个任务和模型架构上持续优于现有最优结果监督型PRMs。代码已公开于https://anonymous.4open.science/r/LCA。

原文摘要 · Abstract (English)

Process reward models (PRMs) enhance the reasoning capabilities of large language models (LLMs) by providing fine-grained feedback, yet training PRMs typically requires expensive stepwise annotations. Outcome-supervised PRMs offer a scalable alternative by learning from final-answer correctness alone, but this introduces a fundamental *credit assignment* challenge, i.e., attributing outcomes to responsible reasoning steps. Existing approaches rely on either uniform or causal assignment, both of which fail to anchor credit in step correctness and thus hinder process error identification. In this work, we propose Outcome-Supervised Process Reward Modeling via **L**earnable **C**redit **A**ssignment (**LCA**), an outcome-supervised PRM framework that jointly learns credit assignment and reward modeling under the principle of *Weakest Link Assignment: a reasoning chain is as strong as its weakest link*. To address mutual dependence between credit assignment and reward modeling, we formalize outcome-supervised PRM as a Multiple Instance Learning (MIL) problem and introduce Softmax-Weighted-Sum (SWS) pooling, an MIL pooling technique tailored for strong dependence and redundancy among reasoning states. We prove Bayes consistency of our algorithm under mild assumptions. Extensive experiments demonstrate that **LCA** consistently outperforms state-of-the-art outcome-supervised PRMs across multiple tasks and backbones. Code is available at https://anonymous.4open.science/r/LCA.

推理增强信用分配奖励建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。