arXiv:2601.03969cs.AIcs.CL2026-01ACL被引 2

通过动态截断冗余推理,让模型更高效地思考。

Anti-Length Shift: Dynamic Outlier Truncation for Training Efficient Reasoning Models

  • 训练时动态识别并裁剪过长的无用推理链。
  • AIME-24上减少78%推理词元,准确率还更高。
  • 适合追求高效推理的模型部署场景。

基于可验证奖励的强化学习使大型推理模型在思维链长度上显著提升性能,但导致简单问题生成冗长回答,带来高昂部署成本。现有方法依赖显式长度惩罚,常引发优化冲突,且未深入分析过度思考的生成机制。本文发现训练中模型对简单输入产生冗余推理的“长度偏移”现象。为此提出训练阶段干预方法动态异常值截断(DOT),仅针对完全正确轨迹中响应长度的极端尾部进行抑制,同时保留复杂问题的长程推理能力。为保障收敛稳定性,引入辅助KL正则化与预测性动态采样。多规模实验表明,该方法显著拓展了效率-性能帕累托前沿。尤其在AIME-24数据集上,推理词元消耗降低78%,准确率高于初始策略,并超越现有最优高效推理方法。

原文摘要 · Abstract (English)

Large reasoning models enhanced by reinforcement learning with verifiable rewards have achieved significant performance gains by extending their chain-of-thought. However, this paradigm incurs substantial deployment costs as models often exhibit excessive verbosity on simple queries. Existing efficient reasoning methods relying on explicit length penalties often introduce optimization conflicts and leave the generative mechanisms driving overthinking largely unexamined. In this paper, we identify a phenomenon termed length shift where models increasingly generate unnecessary reasoning on trivial inputs during training. To address this, we introduce Dynamic Outlier Truncation (DOT), a training-time intervention that selectively suppresses redundant tokens. This method targets only the extreme tail of response lengths within fully correct rollout groups while preserving long-horizon reasoning capabilities for complex problems. To complement this intervention and ensure stable convergence, we further incorporate auxiliary KL regularization and predictive dynamic sampling. Experimental results across multiple model scales demonstrate that our approach significantly pushes the efficiency-performance Pareto frontier outward. Notably, on the AIME-24, our method reduces inference token usage by 78% while simultaneously increasing accuracy compared to the initial policy and surpassing state-of-the-art efficient reasoning methods.

高效推理强化学习思维链模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。