arXiv:2505.11896cs.LGcs.AI2025-05被引 88

让大模型智能决定何时用思维链,省计算又不丢效果。

AdaCoT: Pareto-Optimal Adaptive Chain-of-Thought Triggering via Reinforcement Learning

  • 用强化学习动态调整思维链触发阈值,按需开启。
  • 生产数据测试中思维链使用率降至3.18%,响应词数减少69.06%。
  • 适合追求推理效率与成本平衡的工业级大模型应用。

大型语言模型虽具强大能力,但在复杂推理任务中仍面临挑战。传统思维链(CoT)提示对所有问题无差别生成长推理步骤,导致显著计算开销,尤其对简单输入效率低下。为此,本文提出AdaCoT(自适应思维链),将自适应推理建模为帕累托优化问题,权衡模型性能与思维链调用成本(频率与计算开销)。采用近端策略优化(PPO)的强化学习方法,通过调节惩罚系数动态控制思维链触发边界,使模型根据查询隐含复杂度自主判断是否启用思维链。关键技术创新为选择性损失掩码(SLM),有效防止多阶段强化学习训练中的决策边界坍塌,确保触发机制稳定可靠。实验表明,AdaCoT成功逼近帕累托前沿,在无需复杂推理的查询上大幅降低思维链使用率。例如在生产流量测试集上,思维链触发率最低达3.18%,平均响应词数减少69.06%,同时保持复杂任务的高性能表现。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have demonstrated remarkable capabilities but often face challenges with tasks requiring sophisticated reasoning. While Chain-of-Thought (CoT) prompting significantly enhances reasoning, it indiscriminately generates lengthy reasoning steps for all queries, leading to substantial computational costs and inefficiency, especially for simpler inputs. To address this critical issue, we introduce AdaCoT (Adaptive Chain-of-Thought), a novel framework enabling LLMs to adaptively decide when to invoke CoT. AdaCoT framed adaptive reasoning as a Pareto optimization problem that seeks to balance model performance with the costs associated with CoT invocation (both frequency and computational overhead). We propose a reinforcement learning (RL) based method, specifically utilizing Proximal Policy Optimization (PPO), to dynamically control the CoT triggering decision boundary by adjusting penalty coefficients, thereby allowing the model to determine CoT necessity based on implicit query complexity. A key technical contribution is Selective Loss Masking (SLM), designed to counteract decision boundary collapse during multi-stage RL training, ensuring robust and stable adaptive triggering. Experimental results demonstrate that AdaCoT successfully navigates the Pareto frontier, achieving substantial reductions in CoT usage for queries not requiring elaborate reasoning. For instance, on our production traffic testset, AdaCoT reduced CoT triggering rates to as low as 3.18\% and decreased average response tokens by 69.06%, while maintaining high performance on complex tasks.

思维链强化学习推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。