模型能自动判断问题难易,选择长或短推理路径。
Adaptive Deep Reasoning: Triggering Deep Thinking When Needed
- 根据问题复杂度自动切换长/短链推理。
- 在数学数据集上保持高准确率,不牺牲性能。
- 适合需要高效推理的实时应用场景。
大型语言模型(LLMs)在复杂任务中通过长链推理展现强大能力,但冗长的推理过程显著增加计算成本,制约实际部署。现有方法虽通过提示工程、微调或强化学习缩短推理链,仍需先执行推理。近期工作将长短推理能力融合于单一模型,但依赖人工切换。本文提出一种自主切换机制:先对基础模型进行监督微调以获得长短推理能力,再通过强化学习优化平衡策略——采用长-短自适应分组奖励评估提示复杂度并给予相应奖励;同时引入基于logit的推理模式切换损失,优化初始标记选择,引导推理类型决策。在数学数据集上的实验表明,模型可动态切换推理模式且不显著降低性能,显著提升大模型推理在真实场景中的实用性。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown impressive capabilities in handling complex tasks through long-chain reasoning. However, the extensive reasoning steps involved can significantly increase computational costs, posing challenges for real-world deployment. Recent efforts have focused on optimizing reasoning efficiency by shortening the Chain-of-Thought (CoT) reasoning processes through various approaches, such as length-aware prompt engineering, supervised fine-tuning on CoT data with variable lengths, and reinforcement learning with length penalties. Although these methods effectively reduce reasoning length, they still necessitate an initial reasoning phase. More recent approaches have attempted to integrate long-chain and short-chain reasoning abilities into a single model, yet they still rely on manual control to toggle between short and long CoT. In this work, we propose a novel approach that autonomously switches between short and long reasoning chains based on problem complexity. Our method begins with supervised fine-tuning of the base model to equip both long-chain and short-chain reasoning abilities. We then employ reinforcement learning to further balance short and long CoT generation while maintaining accuracy through two key strategies: first, integrating reinforcement learning with a long-short adaptive group-wise reward strategy to assess prompt complexity and provide corresponding rewards; second, implementing a logit-based reasoning mode switching loss to optimize the model's initial token choice, thereby guiding the selection of the reasoning type. Evaluations on mathematical datasets demonstrate that our model can dynamically switch between long-chain and short-chain reasoning modes without substantially sacrificing performance. This advancement enhances the practicality of reasoning in large language models for real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。