让大模型根据题目难易自动调节思考深度,减少无效推理。
Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoning
- 用显式思维标记让模型思考过程可追踪
- 结合难度估计与字数预算动态切换思考模式
- 适合追求高效推理的AI系统开发者
大型推理模型在复杂任务中表现优异,但常出现过度思考,生成冗余内容。受认知科学双过程理论启发,我们提出自适应认知策略优化(ACPO),一种强化学习框架,使模型通过自适应认知分配和动态系统切换实现高效推理。ACPO包含两个核心组件:(1) 引入系统感知的推理标记,显式表示思考模式,使模型认知过程透明化;(2) 集成在线难度估计与词元长度预算,指导强化学习中的自适应系统切换与推理。为此,我们设计两阶段训练策略:第一阶段通过监督微调冷启动模型,使其生成带有明确思考模式的推理路径;第二阶段应用ACPO进一步增强基于难度感知的自适应系统切换能力。实验表明,ACPO有效减少了冗余推理,并根据任务复杂度自适应调整认知分配,实现了高效的混合推理。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) have demonstrated strong performance on complex reasoning tasks, but often suffer from overthinking, generating redundant content regardless of task difficulty. Inspired by the dual process theory in cognitive science, we propose Adaptive Cognition Policy Optimization (ACPO), a reinforcement learning framework that enables LRMs to achieve efficient reasoning through adaptive cognitive allocation and dynamic system switch. ACPO incorporates two key components: (1) introducing system-aware reasoning tokens to explicitly represent the thinking modes thereby making the model's cognitive process transparent, and (2) integrating online difficulty estimation and token length budget to guide adaptive system switch and reasoning during reinforcement learning. To this end, we propose a two-stage training strategy. The first stage begins with supervised fine-tuning to cold start the model, enabling it to generate reasoning paths with explicit thinking modes. In the second stage, we apply ACPO to further enhance adaptive system switch for difficulty-aware reasoning. Experimental results demonstrate that ACPO effectively reduces redundant reasoning while adaptively adjusting cognitive allocation based on task complexity, achieving efficient hybrid reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。