arXiv:2510.03805cs.CLcs.AI2025-10ACL被引 5

用新奖励机制让大模型少绕弯,又不丢准确性。

Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models

  • 设计步数感知奖励,鼓励简洁推理而非单纯删字
  • 在AIME24上减少69.7%的生成token,准确率仍领先
  • 动态停止机制防模型偷懒合并步骤,适合追求效率的研究者

大型推理模型在复杂任务中表现优异,但常因过度思考导致冗长。现有基于强化学习的方法通过惩罚生成词元来追求简洁,但存在两个问题:词元少未必意味着步骤少,且模型可能在训练后期通过跳过推理步骤来减少消耗,产生“作弊”行为。本文提出步数剪枝器(Step Pruner, SP),一种新的强化学习框架,通过步数感知奖励函数,优先保证正确性,对冗余步骤施加惩罚,并对错误回答不予奖励,防止错误推理被强化。此外,引入动态停止机制:当模型输出不再缩短时即终止训练,避免因合并步骤而引发的作弊行为。在四个推理基准上的实验表明,SP在显著降低响应长度的同时达到最优准确率。例如,在AIME24上,其词元使用量减少了69.7%。

原文摘要 · Abstract (English)

Large Reasoning Models (LRMs) demonstrate strong performance on complex tasks but often suffer from excessive verbosity, known as "overthinking." Existing solutions via reinforcement learning (RL) typically penalize generated tokens to promote conciseness. However, these methods encounter two challenges: responses with fewer tokens do not always correspond to fewer reasoning steps, and models may develop hacking behavior in later stages of training by discarding reasoning steps to minimize token usage. In this work, we introduce \textbf{Step Pruner (SP)}, an RL framework that steers LRMs toward more efficient reasoning by favoring compact reasoning steps. Our step-aware reward function prioritizes correctness while imposing penalties for redundant steps, and withholds rewards for incorrect responses to prevent the reinforcement of erroneous reasoning. Moreover, we propose a dynamic stopping mechanism: when the model's output no longer shortens, training is halted to prevent hacking behavior caused by the merging of steps. Extensive experiments across four reasoning benchmarks demonstrate that SP achieves state-of-the-art accuracy while significantly reducing response length. For instance, on AIME24, SP reduces token usage by \textbf{69.7\%}.

大模型推理强化学习简洁生成推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。