让大模型推理更短更准，自动删冗余步骤
Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning
- 按每步重要性动态分配删减权重，保留关键推理
- 平均减少64.3%推理长度，准确率提升5.7%
- 适合需要高效推理的部署场景
大型推理模型通过增加测试时计算量可提升性能，但常过度思考，产生冗长的思维链，导致成本上升而准确率未增。先前强化学习方法通常仅使用单一结果奖励和轨迹级长度惩罚，无法区分必要与冗余步骤，压缩效果粗糙。尽管近期工作引入了步骤级信号，如离线剪枝、监督数据构建或验证器中间奖励，但推理长度极少作为强化学习中的显式步骤级优化目标。本文提出逐步自适应惩罚（SWAP），一种细粒度框架，根据每步对正确答案的内在贡献分配长度压缩。通过模型在策略上的对数概率提升估计步骤重要性，将多余长度视为惩罚质量，重新分配给低重要性步骤以加重惩罚，同时保留高重要性推理。采用组内相对策略优化中的统一结果-过程优势进行优化。大量实验表明，SWAP平均减少64.3%推理长度，相比基线模型准确率提升5.7%。
原文摘要 · Abstract (English)
Large reasoning models improve with more test-time computation, but often overthink, producing unnecessarily long chains-of-thought that raise cost without improving accuracy. Prior reinforcement learning approaches typically rely on a single outcome reward with trajectory-level length penalties, which cannot distinguish essential from redundant reasoning steps and therefore yield blunt compression. Although recent work incorporates step-level signals, such as offline pruning, supervised data construction, or verifier-based intermediate rewards, reasoning length is rarely treated as an explicit step-level optimization objective during RL. We propose Step-wise Adaptive Penalization (SWAP), a fine-grained framework that allocates length reduction across steps based on intrinsic contribution. We estimate step importance from the model's on-policy log-probability improvement toward the correct answer, then treat excess length as a penalty mass redistributed to penalize low-importance steps more heavily while preserving high-importance reasoning. We optimize with a unified outcome-process advantage within group-relative policy optimization. Extensive experiments demonstrate that SWAP reduces reasoning length by 64.3% on average while improving accuracy by 5.7% relative to the base model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。