arXiv:2505.21097cs.CLcs.AI2025-05NeurIPS被引 10

让大模型分快慢思维答题,提升准确率与效率。

Thinker: Learning to Think Fast and Slow

  • 设计四阶段问答流程:快速作答、验证、慢速修正、精炼总结。
  • Qwen2.5-1.5B准确率从25.6%提至27.3%,DeepSeek-R1-Qwen-1.5B从45.9%提至51.0%。
  • 快思模式仅用不足1000词达25.2%准确率,推理更高效,适合资源受限场景。

近期研究显示,通过强化学习(RL)训练大语言模型(LLM)在数学和编程等领域的问答任务中,可提升其推理能力。长上下文下,LLM可能表现出搜索行为,如DeepSeek R1中的自我修正现象,但此类行为常不精准且缺乏信心,导致响应冗长重复,暴露出直觉与验证能力的不足。受心理学双过程理论启发,我们提出一种包含四个阶段的问答任务改进:快速思考(在严格词元预算内作答)、验证(评估初始答案)、慢速思考(进一步推敲修正)、总结(提炼为精确步骤)。该方法使Qwen2.5-1.5B平均准确率从25.6%提升至27.3%,DeepSeek-R1-Qwen-1.5B从45.9%提升至51.0%。值得注意的是,对Qwen2.5-1.5B而言,仅使用快速思考模式即达25.2%准确率,且耗时少于1000词,展现出显著的推理效率优势。结果表明,直觉与深思是独立互补的系统,需针对性训练。相关模型与代码已开源。

原文摘要 · Abstract (English)

Recent studies show that the reasoning capabilities of Large Language Models (LLMs) can be improved by applying Reinforcement Learning (RL) to question-answering (QA) tasks in areas such as math and coding. With a long context length, LLMs may learn to perform search, as indicated by the self-correction behavior observed in DeepSeek R1. However, this search behavior is often imprecise and lacks confidence, resulting in long, redundant responses and highlighting deficiencies in intuition and verification. Inspired by the Dual Process Theory in psychology, we introduce a simple modification to the QA task that includes four stages: Fast Thinking, where the LLM must answer within a strict token budget; Verification, where the model evaluates its initial response; Slow Thinking, where it refines the initial response with more deliberation; and Summarization, where it distills the refinement from the previous stage into precise steps. Our proposed task improves average accuracy from 25.6% to 27.3% for Qwen2.5-1.5B, and from 45.9% to 51.0% for DeepSeek-R1-Qwen-1.5B. Notably, for Qwen2.5-1.5B, the Fast Thinking mode alone achieves 25.2% accuracy using fewer than 1000 tokens, demonstrating substantial inference efficiency gains. These findings suggest that intuition and deliberative reasoning are distinct, complementary systems benefiting from targeted training. Additionally, we have open-sourced both the trained models and the source code.

大模型推理双过程理论强化学习效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。