arXiv:2505.13417cs.CLcs.AI2025-05EMNLP被引 159

让大模型学会根据题目难易自动选择思考或不思考,兼顾效率与准确率。

AdaptThink: Reasoning Models Can Learn When to Think

  • 用强化学习教模型自适应选择是否思考,提升推理效率。
  • 在三个数学数据集上减少53%响应长度,准确率提升2.4%。
  • 适合追求高效推理的AI应用,如实时问答系统。

近期大型推理模型通过模拟人类深度思考,在各类任务中取得优异表现,但冗长的思考过程显著增加了推理开销,效率成为关键瓶颈。本文首次证明,对较简单任务采用'无思考'(NoThinking)策略——直接生成答案而不进行推理——在性能与效率上均更优。受此启发,我们提出AdaptThink,一种新型强化学习算法,使推理模型能根据问题难度自适应选择最优思考模式。核心包含两个组件:(1) 约束优化目标,鼓励模型在保持整体性能前提下优先选择无思考;(2) 重要性采样策略,在在线训练中平衡有思考与无思考样本,支持冷启动并促进模型持续探索与利用两种模式。实验表明,AdaptThink显著降低推理成本并进一步提升性能。特别地,在三个数学数据集上,该方法将DeepSeek-R1-Distill-Qwen-1.5B的平均响应长度减少53%,准确率提升2.4%,凸显了自适应思考模式选择在推理质量与效率间优化的巨大潜力。代码与模型已开源。

原文摘要 · Abstract (English)

Recently, large reasoning models have achieved impressive performance on various tasks by employing human-like deep thinking. However, the lengthy thinking process substantially increases inference overhead, making efficiency a critical bottleneck. In this work, we first demonstrate that NoThinking, which prompts the reasoning model to skip thinking and directly generate the final solution, is a better choice for relatively simple tasks in terms of both performance and efficiency. Motivated by this, we propose AdaptThink, a novel RL algorithm to teach reasoning models to choose the optimal thinking mode adaptively based on problem difficulty. Specifically, AdaptThink features two core components: (1) a constrained optimization objective that encourages the model to choose NoThinking while maintaining the overall performance; (2) an importance sampling strategy that balances Thinking and NoThinking samples during on-policy training, thereby enabling cold start and allowing the model to explore and exploit both thinking modes throughout the training process. Our experiments indicate that AdaptThink significantly reduces the inference costs while further enhancing performance. Notably, on three math datasets, AdaptThink reduces the average response length of DeepSeek-R1-Distill-Qwen-1.5B by 53% and improves its accuracy by 2.4%, highlighting the promise of adaptive thinking-mode selection for optimizing the balance between reasoning quality and efficiency. Our codes and models are available at https://github.com/THU-KEG/AdaptThink.

推理优化强化学习自适应推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。