让大模型动态选工具,推理更灵活,性能提升显著。
AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning
- 训练模型动态选择工具,结合强化学习与排序优化。
- 在数学、代码等任务上平均提升6.4%至7.7%。
- 能用未见过的工具,适合需要自适应能力的场景。
代理式强化学习已使大语言模型(LLMs)能够通过长链式思维轨迹进行推理,并穿插使用外部工具。现有方法假设工具集固定,限制了模型对新或演化工具集的适应能力。我们提出AutoTool,一种训练框架,使LLM代理在推理过程中具备动态工具选择能力。AutoTool采用双阶段优化流程:(i) 基于SFT和强化学习的轨迹稳定性优化,保证推理连贯性;(ii) KL正则化的Plackett-Luce排序,优化多步工具选择的一致性。我们构建了一个包含20万条样本的数据集,覆盖1000+工具和100+任务,涵盖数学、科学、代码生成及多模态推理。在十项基准测试中,使用Qwen3-8B和Qwen2.5-VL-7B两个基础模型进行训练。参数更少的情况下,AutoTool在数学与科学推理上平均提升6.4%,搜索型问答提升4.5%,代码生成提升7.7%,多模态理解提升6.9%。此外,AutoTool在推理时可动态调用未见工具,表现出更强泛化能力。
原文摘要 · Abstract (English)
Agentic reinforcement learning has advanced large language models (LLMs) to reason through long chain-of-thought trajectories while interleaving external tool use. Existing approaches assume a fixed inventory of tools, which limits the adaptability of LLM agents to new or evolving toolsets. We present AutoTool, a training framework that equips LLM agents with dynamic tool-selection capabilities throughout their reasoning trajectories. AutoTool employs a dual-phase optimization pipeline: (i) SFT and RL-based trajectory stabilization for coherent reasoning, and (ii) KL-regularized Plackett-Luce Ranking to refine consistent multi-step tool selection. We further build a 200k dataset with explicit tool-selection rationales across 1,000+ tools and 100+ tasks spanning mathematics, science, code generation, and multimodal reasoning. Across ten diverse benchmarks, we train two base models, Qwen3-8B and Qwen2.5-VL-7B, with AutoTool. With fewer parameters, AutoTool consistently outperforms advanced LLM agents and tool-integration methods, yielding average gains of 6.4% in math & science reasoning, 4.5% in search-based QA, 7.7% in code generation, and 6.9% in multimodal understanding. In addition, AutoTool exhibits stronger generalization by dynamically leveraging unseen tools from evolving toolsets during inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。