让推理模型自动学会恰到好处的思考长度，又快又准。
ShorterBetter: Guiding Reasoning Models to Find Optimal Inference Length for Efficient Reasoning
- 用强化学习让模型自己学最优思考长度，无需人工标注。
- 在多个任务上减少50%-80%输出长度，准确率不变。
- 能消除重复、过度验证和无效探索，让推理更干净。
近期如 OpenAI o1 和 DeepSeek-R1 等模型通过生成较长的思维链（Chain-of-Thought, CoT）在复杂推理任务中表现优异。然而，过长的推理常导致冗余和低效，即所谓“过度思考”。本文提出 ShorterBetter，一种简单有效的强化学习方法，使推理模型能自主学习最优的 CoT 长度，无需人工监督。我们定义样本最优长度（Sample Optimal Length, SOL）为多次生成中首个正确答案的最短长度，作为动态奖励信号，引导模型高效推理。在 DeepSeek-Distill-Qwen-1.5B/7B 上应用后，ShorterBetter 在域内与域外推理任务中均实现 50%–80% 的输出长度缩减，同时保持准确率。分析显示，该方法通过减少重复、过度自我验证和对备选路径的冗余探索，优化了推理结构。
原文摘要 · Abstract (English)
Recent models such as OpenAI o1 and DeepSeek-R1 have demonstrated strong performance on reasoning-intensive tasks by generating extended Chain-of-Thought (CoT) traces. While longer reasoning helps with thorough exploration of solution paths for complex problems, it also often leads to inefficient and redundant outputs--a phenomenon commonly described as overthinking. In this paper, we propose ShorterBetter, a simple yet effective reinforcement learning method that enables reasoning models to learn their own optimal CoT lengths without manual supervision. We define the Sample Optimal Length (SOL) as the length of the shortest correct response among multiple generations, which serves as a dynamic reward signal to guide the model toward efficient reasoning. Applied to DeepSeek-Distill-Qwen-1.5B/7B as base models, ShorterBetter achieves 50%-80% reduction in output lengths in both in-domain and out-of-domain reasoning tasks while maintaining accuracy. Our reasoning trace analysis shows that ShorterBetter refines the structure of the reasoning traces by reducing unnecessary repetition, excessive self-verification, and over-exploration of alternatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。