提出新评估方法，让大模型少思考、多省资源。
The Price of a Second Thought: On the Evaluation of Reasoning Efficiency in Large Language Models
- 用指令模型作基线，衡量思维模型的推理效率
- 发现复杂问题才需深度思考，简单题易过度推理
- 设计两阶段流程，节省21.1% token仍保精度
近期基于强化学习和反向检查链式思维（CoT）训练的思维模型常出现过度思考：在简单问题上生成过长输出，浪费计算资源。现有基于令牌效率的评估方法因忽略问题难度和中间计算成本，无法全面反映真实效率。本文将推理效率定义为思维模型与指令模型之间的相对度量，以指令模型作为最低努力基准。在四个思维模型与多个基准上的系统研究揭示两个一致规律：(i) 指令模型整体效率更高；(ii) 问题难度影响效率，思维模型在简单问题上浪费计算，但在难题中仍具价值。基于此，我们提出COTHINK，一种简单两阶段流程：先由指令模型起草简要提纲，再由思维模型展开。在GSM8K、MATH500和AIME24上，COTHINK将令牌使用量减少21.1%，同时保持四种思维模型的准确率，并优于强效率基线。
原文摘要 · Abstract (English)
Recent thinking models trained with reinforcement learning and backward-checking CoT often suffer from overthinking: they produce excessively long outputs even on simple problems, wasting computation. Existing evaluations, based on token efficiency, give an incomplete view as they neglect problem difficulty and intermediate computation costs. We formalize reasoning efficiency as a relative measure between thinking and instruct models, treating instruct models as the minimal-effort baseline. A systematic study across four thinking models and multiple benchmarks reveals two consistent patterns: (i) instruct models achieve higher efficiency overall, and (ii) problem difficulty affects efficiency, with thinking models wasting computation on easy problems but providing value on harder ones. Building on this insight, we propose COTHINK, a simple two-stage pipeline: an instruct model drafts a brief outline, and a thinking model expands it. On GSM8K, MATH500, and AIME24, COTHINK cuts token usage by 21.1% while keeping accuracy on four thinking models, and remains competitive with strong efficiency baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。