arXiv:2507.04348cs.AIcs.CL2025-07被引 10

让推理模型按步骤重要性自动调节长度,既省计算又不丢关键逻辑。

SmartThinker: Learning to Compress and Preserve Reasoning by Step-Level Length Control

  • 按每一步重要性动态控制推理长度,避免一刀切压缩。
  • 在多个基准上减少冗余推理,准确率保持甚至提升。
  • 适合追求高效推理的模型部署与资源受限场景。

大型推理模型(LRMs)通过推理时扩展展现出强大推理能力,但也带来了显著冗余和效率低下问题,造成大量计算浪费。以往工作通过强化学习中对生成样本整体长度施加惩罚来缓解,但常导致关键步骤被过度压缩,而简单步骤仍保留冗余信息,难以平衡准确率与效率。为此,我们提出 SmartThinker,一个两阶段可学习框架,实现基于步骤重要性的细粒度推理链长度控制。第一阶段,通过拒绝采样结合监督微调(SFT),将模型适配为短形式推理模式;第二阶段,采用步级长度控制策略优化(SCPO),调整模型输出分布,增加关键步骤的长度分配,减少非关键步骤冗余。SCPO包含四个核心组件:在线重要性估计算器、步级长度控制奖励函数、步级广义优势估计(S-GAE)及难度自适应截断策略,协同实现跨步骤差异化长度控制。在多个推理基准和多种骨干模型上的实验表明,SmartThinker显著减少冗余推理,同时达到或超越现有方法的性能。

原文摘要 · Abstract (English)

Large reasoning models (LRMs) have exhibited remarkable reasoning capabilities through inference-time scaling, but this progress has also introduced considerable redundancy and inefficiency into their reasoning processes, resulting in substantial computational waste. Previous work has attempted to mitigate this issue by penalizing the overall length of generated samples during reinforcement learning (RL), with the goal of encouraging a more concise chains of thought. However, we observe that such global length penalty often lead to excessive compression of critical reasoning steps while preserving unnecessary details in simpler ones, yielding a suboptimal trade-off between accuracy and efficiency. To address this issue, we propose SmartThinker, a two-stage learnable framework designed to enable fine-grained control over the length of reasoning chains based on the importance of each individual step. In the first stage, SmartThinker adapts a reasoning model to a short-form reasoning mode through rejection sampling combined with supervised fine-tuning (SFT). In the second stage, SmartThinker applies Step-Level Length Control Policy Optimization (SCPO) to refine the model output distribution, which increases the proportion of length allocated to critical steps while reducing redundancy in less important ones. SCPO consists of four core components: an online importance estimator, a step-level length control reward function, a step-level generalized advantage estimation (S-GAE) and a difficulty-adaptive clipping strategy. Working in concert, these components enable SCPO to implement differentiated length control across reasoning steps. Empirical results across multiple reasoning benchmarks and various backbone models demonstrate that SmartThinker significantly reduces redundant reasoning while achieving comparable or even superior performance to existing methods.

推理压缩强化学习模型效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。