让模型自己推理每步逻辑是否正确，提升复杂问题求解能力
StepWiser: Stepwise Generative Judges for Wiser Reasoning
- 用生成式裁判逐步分析推理过程，输出思考路径再判对错
- 相比旧方法，中间步骤判断准确率更高，且能反哺策略模型训练
- 适合需要可靠逻辑验证的AI推理系统研发者
随着模型越来越多地采用多步推理解决复杂问题，监督中间步骤的逻辑有效性成为关键挑战。过程奖励模型通过逐步反馈解决此问题，但现有方法存在两大缺陷：通常作为无解释的分类器，且依赖静态数据集的监督微调，限制了泛化能力。受最新进展启发，我们把步骤奖励建模从分类任务重构为推理任务本身。为此提出生成式裁判StepWiser，它对策略模型的推理步骤进行元推理（即‘思考如何思考’），先输出思维标记，再给出最终判断。该模型通过强化学习训练，利用回溯展开的相对结果优化。实验表明，StepWiser在中间步骤判断上优于现有方法；可应用于训练时提升策略模型；并能改进推理阶段的搜索性能。
原文摘要 · Abstract (English)
As models increasingly leverage multi-step reasoning strategies to solve complex problems, supervising the logical validity of these intermediate steps has become a critical research challenge. Process reward models address this by providing step-by-step feedback, but current approaches have two major drawbacks: they typically function as classifiers without providing explanations, and their reliance on supervised fine-tuning with static datasets limits generalization. Inspired by recent advances, we reframe stepwise reward modeling from a classification task to a reasoning task itself. We thus propose a generative judge that reasons about the policy model's reasoning steps (i.e., meta-reasons), outputting thinking tokens before delivering a final verdict. Our model, StepWiser, is trained by reinforcement learning using relative outcomes of rollouts. We show it provides (i) better judgment accuracy on intermediate steps than existing methods; (ii) can be used to improve the policy model at training time; and (iii) improves inference-time search.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。