arXiv:2609.04648cs.CL2026-09

通过共识节点提升大模型长链推理的奖励密度,让中间步骤也能被有效反馈。

ConsensusBench: Benchmark of Consensus Nodes for LLM Reasoning via Outcome Reward Densifying

论文配图:ConsensusBench: Benchmark of Consensus Nodes for LLM Reasoning via Outcome Reward Densifying
图 1 · 摘自论文原文
  • 基于正确路径聚类生成可验证的中间结论节点作为过程信号
  • 在多个数据集上比传统方法提升推理准确率,尤其在长链条任务中优势明显
  • 适合关注长链推理优化、强化学习奖励设计的研究者

强化学习已成为提升大语言模型推理能力的主要范式。其中,组相对策略优化(GRPO)等算法在基于最终答案的奖励下表现优异。然而,这些方法仅依赖最终结果,缺乏对中间步骤贡献的反馈。随着任务复杂度和推理路径长度增加,这种稀疏的最终奖励变得愈发不足。为此,我们提出 ConsensusBench,一个新型数据集,旨在提供基于规则的过程级信号。我们假设正确最终答案依赖于推理过程中少数关键的中间结论,这些可视为可验证的子结果。通过从N次采样轨迹中筛选正确路径,并对语义等价的中间陈述进行聚类,我们定义这些聚类结果为共识节点(Consensus Nodes)。将基于这些节点的规则化过程奖励引入GRPO类算法,构建出新的强化学习信号——ConsensusPR,显著降低长推理路径中的奖励稀疏性。为系统评估过程级表现,我们引入三项指标:最终答案准确率(Acc)、节点覆盖率(NCR)和每节点词数(TPN)。在AIME 2024、AIME 2025、GSM8K、MATH-500及我们的ConsensusBench上的实验表明,该方法持续优于传统GRPO类方法,证明了共识节点在引导推理中的实际价值。

原文摘要 · Abstract (English)

Reinforcement learning (RL) has become one of the primary paradigms for reasoning enhancement of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) and related algorithms have demonstrated strong performance with outcome-level rewards. However, these methods depend solely on the final answer, without feedback regarding which intermediate steps contribute to success or failure. As task complexity and reasoning trajectory length increase, such sparse final-answer rewards become increasingly insufficient. To address this limitation, we introduce ConsensusBench, a novel dataset designed to provide rule-based process-level signals. We posit that a correct final answer relies on a small set of intermediate conclusions throughout the reasoning process, which can be seen as a verifiable sub-outcome. We identify these sub-outcomes by filtering correct trajectories from N rollouts and clustering semantically equivalent intermediate statements. We call these clustered statements as Consensus Nodes. By integrating a rule-based process reward derived from these nodes into GRPO-style algorithms, we develop a new reinforcement learning signal named ConsensusPR. It directly reduces the reward sparsity of outcome reward across long reasoning trajectories. To facilitate systematic process-level evaluation, we introduce three metrics to our benchmark: Final Answer Accuracy (Acc), Node Coverage Rate (NCR), and Tokens per Node (TPN). Experiments across AIME 2024, AIME 2025, GSM8K, MATH-500, and our ConsensusBench demonstrate that the proposed method consistently surpasses GRPO-style approaches, highlighting the practical value of consensus nodes in guiding reasoning.

大模型推理强化学习过程奖励共识节点

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。