让大模型在长链条推理中自动发现并使用工具，无需人工标注。
Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees
- 构建动态回溯树，自动探测工具调用时机
- 在AIME和GPQA-Diamond上显著超越现有方法
- 适合需要复杂推理与计算结合的任务场景
工具集成推理已成为增强大语言模型计算能力的关键范式，但将工具使用融入长链思维（long CoT）仍缺乏探索，主要受限于训练数据稀缺及工具使用对模型内在长链推理能力的干扰。本文提出DART（基于回溯树的工具集成推理发现与强化框架），通过在训练过程中构建动态回溯树，自动发现有效的工具调用机会，在有潜力的位置分支探索多样化的工具融合路径。随后，基于树结构的优势估计过程识别并奖励那些工具调用对解题产生正向贡献的子轨迹，从而有效强化有益行为。在AIME和GPQA-Diamond等高难度基准上的大量实验表明，DART显著优于现有方法，成功实现工具执行与长链推理的协同优化。
原文摘要 · Abstract (English)
Tool-Integrated Reasoning has emerged as a key paradigm to augment Large Language Models (LLMs) with computational capabilities, yet integrating tool-use into long Chain-of-Thought (long CoT) remains underexplored, largely due to the scarcity of training data and the challenge of integrating tool-use without compromising the model's intrinsic long-chain reasoning. In this paper, we introduce DART (Discovery And Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees), a reinforcement learning framework that enables spontaneous tool-use during long CoT reasoning without human annotation. DART operates by constructing dynamic rollout trees during training to discover valid tool-use opportunities, branching out at promising positions to explore diverse tool-integrated trajectories. Subsequently, a tree-based process advantage estimation identifies and credits specific sub-trajectories where tool invocation positively contributes to the solution, effectively reinforcing these beneficial behaviors. Extensive experiments on challenging benchmarks like AIME and GPQA-Diamond demonstrate that DART significantly outperforms existing methods, successfully harmonizing tool execution with long CoT reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。