让大模型学会适时停止思考，提升推理效率
Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking
- 通过轨迹截断训练模型提前终止冗余推理
- 在奥数基准上准确率提升4.6%，输出长度减少46.3%
- 适合追求高效推理的模型部署场景
大型推理模型（LRMs）在复杂任务上表现优异，但深度推理带来高昂计算开销。现有强化学习方法在推演阶段仍难以生成简洁推理路径，限制了有效学习。受证据累积模型启发，我们发现模型在推理早期已积累足够信息，后续步骤冗余。为此提出Just-Enough Thinking（JET），训练模型主动终止不必要的推理。JET在推演中执行轨迹截断，使模型接触短而分布一致的推理路径；同时使用质量可控的长度奖励，鼓励简洁推理并保持正确性。大量实验表明，JET显著提升推理效率且不损失准确性。尤其在Olympiad基准上，DeepSeek-Distill-Qwen-1.5B实现4.6%准确率提升，输出长度降低46.3%。代码已开源。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) have achieved impressive performance on challenging tasks, yet their deep reasoning often incurs substantial computational costs. To achieve efficient reasoning, existing reinforcement learning methods still struggle to construct short reasoning path during the rollout stage, limiting effective learning. Inspired by Evidence Accumulation Models, we find that LRMs have accumulated sufficient information early in reasoning, making further reasoning steps redundant. Based on this insight, we propose Just-Enough Thinking (JET), which trains models to proactively terminate unnecessary reasoning. JET performs trajectory truncation during rollout to expose the model to short, distributionally consistent reasoning paths. Besides, it uses a quality-controlled length reward to better encourage concise reasoning while maintaining correctness. Extensive experiments demonstrate that JET significantly improves reasoning efficiency without sacrificing accuracy. Especially, DeepSeek-Distill-Qwen-1.5B achieves a 4.6% accuracy gain while reducing output length by 46.3% on the Olympiad benchmark. Our code is available in the GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。