arXiv:2505.10832cs.CLcs.AI2025-05NeurIPS被引 65

让大模型学会判断何时该思考,自动选择推理或直接作答。

Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RL

  • 用多阶段强化学习训练模型,根据问题难易动态决定是否推理。
  • 在5个数学基准上提升6.4%准确率,同时减少52%的 token 消耗。
  • 可无缝接入各类R1风格模型,适合追求效率与精度平衡的场景。

大型推理模型(LRMs)擅长生成详细的分步推理过程以得出最终答案,但这种冗长推理会带来显著的计算开销和延迟,尤其对简单问题而言。为解决过度推理问题,本文探索如何赋予LRMs自适应思考能力:根据问题复杂度动态决定是否进行显式推理。基于R1风格的蒸馏模型,我们发现向提示中插入一个省略号(...)可随机触发思考或不思考模式,揭示了推理行为中的潜在可控性。据此提出AutoThink,一种多阶段强化学习框架,通过分阶段奖励设计逐步优化推理策略。AutoThink仅在必要时调用显式推理,简单任务则默认生成简洁回答。在五个主流数学基准上的实验表明,相比近期提示工程与基于RL的剪枝方法,AutoThink实现了更优的准确性-效率权衡。该方法可无缝集成至任意R1风格模型,包括蒸馏版和后续微调版本。特别地,在DeepSeek-R1-Distill-Qwen-1.5B上,相对准确率提升6.4%,token使用量降低52%,建立了一种可扩展的自适应推理范式。

原文摘要 · Abstract (English)

Large reasoning models (LRMs) are proficient at generating explicit, step-by-step reasoning sequences before producing final answers. However, such detailed reasoning can introduce substantial computational overhead and latency, particularly for simple problems. To address this over-thinking problem, we explore how to equip LRMs with adaptive thinking capabilities: enabling them to dynamically decide whether or not to engage in explicit reasoning based on problem complexity. Building on R1-style distilled models, we observe that inserting a simple ellipsis ("...") into the prompt can stochastically trigger either a thinking or no-thinking mode, revealing a latent controllability in the reasoning behavior. Leveraging this property, we propose AutoThink, a multi-stage reinforcement learning (RL) framework that progressively optimizes reasoning policies via stage-wise reward shaping. AutoThink learns to invoke explicit reasoning only when necessary, while defaulting to succinct responses for simpler tasks. Experiments on five mainstream mathematical benchmarks demonstrate that AutoThink achieves favorable accuracy-efficiency trade-offs compared to recent prompting and RL-based pruning methods. It can be seamlessly integrated into any R1-style model, including both distilled and further fine-tuned variants. Notably, AutoThink improves relative accuracy by 6.4 percent while reducing token usage by 52 percent on DeepSeek-R1-Distill-Qwen-1.5B, establishing a scalable and adaptive reasoning paradigm for LRMs. Project Page: https://github.com/ScienceOne-AI/AutoThink.

自适应推理强化学习效率优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。