通过中间置信度与正确性构建步骤潜力信号,实现数学推理的精细奖励分配。
Step Potential Advantage Estimation: Harnessing Intermediate Confidence and Correctness for Efficient Mathematical Reasoning
- 引入步骤潜力信号,量化每步推理状态,支持细粒度信用分配。
- 在多个基准上提升准确率,响应长度减少30%以上,优于主流强化学习方法。
- 无需训练,适用于任何大模型,适合需要高效精准推理的场景。
基于可验证奖励的强化学习(RLVR)能激发大语言模型(LLM)产生长链思维过程,但基于结果的奖励导致优势估计粗糙。现有方法虽通过词元级熵或序列长度控制改进RLVR,却缺乏语义层面的步骤级推理进展度量。因此,模型无法区分必要推导与冗余验证:可能在得出正确答案后仍继续检查,甚至将正确路径错误逆转。为弥补过程监督缺失,我们提出一种无需训练的探针机制,提取中间置信度与正确性,并融合成步骤潜力信号,显式估计每一步的推理状态。基于此信号,我们设计步骤潜力优势估计(SPAE),通过放大潜在收益、惩罚潜在下降,并在潜力饱和后施加惩罚,促使及时终止。多基准实验表明,SPAE持续提升准确率,显著缩短响应长度,优于强基线及近期高效推理与词元级优势估计方法。代码已开源:https://github.com/cii030/SPAE-RL。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) elicits long chain-of-thought reasoning in large language models (LLMs), but outcome-based rewards lead to coarse-grained advantage estimation. While existing approaches improve RLVR via token-level entropy or sequence-level length control, they lack a semantically grounded, step-level measure of reasoning progress. As a result, LLMs fail to distinguish necessary deduction from redundant verification: they may continue checking after reaching a correct solution and, in extreme cases, overturn a correct trajectory into an incorrect final answer. To remedy the lack of process supervision, we introduce a training-free probing mechanism that extracts intermediate confidence and correctness and combines them into a Step Potential signal that explicitly estimates the reasoning state at each step. Building on this signal, we propose Step Potential Advantage Estimation (SPAE), a fine-grained credit assignment method that amplifies potential gains, penalizes potential drops, and applies penalty after potential saturates to encourage timely termination. Experiments across multiple benchmarks show SPAE consistently improves accuracy while substantially reducing response length, outperforming strong RL baselines and recent efficient reasoning and token-level advantage estimation methods. The code is available at https://github.com/cii030/SPAE-RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。