arXiv:2604.05164cs.LGcs.AI2026-04被引 3

让大模型按难易程度分配思考资源,省时省力还更准。

Not All Turns Are Equally Hard: Adaptive Thinking Budgets For Efficient Multi-Turn Reasoning in Agents

  • 根据每轮对话难易动态分配计算预算
  • 最多节省35%令牌,延迟降低30%
  • 适合需要高效多轮推理的智能体系统

随着大模型推理性能趋于饱和,提升推理阶段的计算效率对缓解过度思考和冗长推理链至关重要。现有方法如长度正则化、自适应路由和难度驱动的预算分配主要针对单轮场景,未能解决多轮推理中的序列依赖问题。本文将多轮推理建模为一个序列计算分配问题,并将其形式化为多目标马尔可夫决策过程。提出TAB(Turn-Adaptive Budgets)策略,通过分组相对策略优化(GRPO)训练,学习在满足全局每题令牌约束的前提下最大化任务准确率。TAB接收对话历史输入,自适应地为简单轮次分配较少预算,为关键难点保留足够令牌。在多个代理基准测试中,TAB在保持准确率的同时,相比静态及现成基线模型,最多节省35%令牌,降低30%延迟。此外,当所有轮次计划可预先获得时,提出TAB All-SubQ,基于对话历史与完整计划分配预算,最多比基线节省40%令牌。

原文摘要 · Abstract (English)

As LLM reasoning performance plateaus, improving inference-time compute efficiency is crucial to mitigate overthinking and long thinking traces even for simple queries. Prior approaches including length regularization, adaptive routing, and difficulty-based budget allocation primarily focus on single-turn settings and fail to address the sequential dependencies inherent in multi-turn reasoning. In this work, we formulate multi-turn reasoning as a sequential compute allocation problem and model it as a multi-objective Markov Decision Process. We propose TAB: Turn-Adaptive Budgets, a budget allocation policy trained via Group Relative Policy Optimization (GRPO) that learns to maximize task accuracy while respecting global per-problem token constraints. Consequently, TAB takes as input the conversation history and learns to adaptively allocate smaller budgets to easier turns and save appropriate number of tokens for the crucial harder reasoning steps. Our experiments on diverse agentic benchmarks demonstrate that TAB achieves a superior accuracy-cost tradeoff saving up to 35% tokens and reducing latency up to 30% while maintaining accuracy over static and off-the-shelf LLM budget baselines. Further, for systems where a plan of all turns is available apriori, we propose TAB All-SubQ, a budget allocation policy that budgets tokens based on the conversation history and entire plan saving up to 40% tokens over baselines.

多轮推理计算效率预算分配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。