arXiv:2508.10293cs.AI2025-08AAAI被引 26

用可验证的逐步奖励机制,让大模型少想多准。

Promoting Efficient Reasoning with Verifiable Stepwise Reward

  • 设计可验证的逐步奖励机制,按中间步骤表现打分
  • 在AIME24/25上减少输出长度,准确率不变
  • 适合需要高效推理的数学与逻辑任务

大型推理模型在复杂推理任务中取得显著进展,得益于基于可验证奖励的强化学习。然而,这些模型常出现过度思考问题,对简单任务消耗过多计算资源,降低效率。现有高效推理方法通常需精准评估任务以预设令牌预算或选择推理模式,限制了灵活性和可靠性。本文重新审视过度思考的本质,发现鼓励有效步骤、惩罚无效步骤是关键。为此,提出一种基于规则的可验证逐步奖励机制(VSRM),根据推理轨迹中各中间状态的表现分配奖励。该方法直观且自然契合逐步推理任务特性。在标准数学推理基准(包括AIME24和AIME25)上,将VSRM与PPO和Reinforce++结合进行大量实验。结果表明,该方法在保持原有推理性能的前提下,大幅减少输出长度,实现效率与准确性的最优平衡。进一步分析显示,训练后无效步骤频率下降,pass@k分数提升,有效抑制了无效推理,从根本上缓解了过度思考问题。代码将在接受后公开。

原文摘要 · Abstract (English)

Large reasoning models (LRMs) have recently achieved significant progress in complex reasoning tasks, aided by reinforcement learning with verifiable rewards. However, LRMs often suffer from overthinking, expending excessive computation on simple problems and reducing efficiency. Existing efficient reasoning methods typically require accurate task assessment to preset token budgets or select reasoning modes, which limits their flexibility and reliability. In this work, we revisit the essence of overthinking and identify that encouraging effective steps while penalizing ineffective ones is key to its solution. To this end, we propose a novel rule-based verifiable stepwise reward mechanism (VSRM), which assigns rewards based on the performance of intermediate states in the reasoning trajectory. This approach is intuitive and naturally fits the step-by-step nature of reasoning tasks. We conduct extensive experiments on standard mathematical reasoning benchmarks, including AIME24 and AIME25, by integrating VSRM with PPO and Reinforce++. Results show that our method achieves substantial output length reduction while maintaining original reasoning performance, striking an optimal balance between efficiency and accuracy. Further analysis of overthinking frequency and pass@k score before and after training demonstrates that our approach in deed effectively suppresses ineffective steps and encourages effective reasoning, fundamentally alleviating the overthinking problem. All code will be released upon acceptance.

推理优化强化学习数学推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。