首次证明工具增强让大模型突破推理瓶颈,解锁新解题能力。
Understanding Tool-Integrated Reasoning
- 用形式化证明工具扩展大模型的可解问题范围。
- 在数学基准上,工具模型在pass@k指标上显著超越纯文本模型。
- 发现模型通过早期调用代码实现更智能的工具使用,适合研究强化学习与模型推理。
我们研究为何工具集成推理(TIR)能提升大语言模型(LLM)的能力。尽管集成如Python解释器等工具的LLM展现出巨大潜力,但其有效性的系统性理论仍缺失。本文首次提供形式化证明,表明TIR从根本上扩展了模型的实证与可行支持集,打破纯文本模型的能力天花板,使原本不可行或冗长复杂的策略成为可能。为在不损害训练稳定性和性能的前提下引导模型行为,我们提出优势塑造策略优化(ASPO),直接修改优势函数以指导策略。在数学难题基准上,利用Python解释器作为外部工具进行综合实验,结果表明TIR模型在pass@k指标上明显优于纯文本模型。关键的是,该优势不仅限于计算密集型问题,也涵盖需深度抽象洞察的任务。我们进一步识别出模型学习‘用工具思考’的涌现认知模式。此外,使用ASPO后,模型更早调用代码,交互轮次显著增加。总体而言,本工作首次为TIR的成功提供了原则性解释,将关注点从‘工具是否有效’转向‘为何且如何实现更强推理’。
原文摘要 · Abstract (English)
We study why Tool-Integrated Reasoning (TIR) makes Large Language Models (LLMs) more capable. While LLMs integrated with tools like Python code interpreters show great promise, a principled theory explaining why this paradigm is effective has been missing. This work provides the first formal proof that TIR fundamentally expands an LLM's capabilities. We demonstrate that tools enable a strict expansion of the model's empirical and feasible support, breaking the capability ceiling of pure-text models by unlocking problem-solving strategies that are otherwise impossible or intractably verbose. To guide model behavior without compromising training stability and performance, we also introduce Advantage Shaping Policy Optimization (ASPO), a novel algorithm that directly modifies the advantage function to guide the policy behavior. We conduct comprehensive experiments on challenging mathematical benchmarks, leveraging a Python interpreter as the external tool. Our results show that the TIR model decisively outperforms its pure-text counterpart on the pass@k metric. Crucially, this advantage is not confined to computationally-intensive problems but extends to those requiring significant abstract insight. We further identify the emergent cognitive patterns that illustrate how models learn to think with tools. Finally, we report improved tool usage behavior with early code invocation and much more interactive turns with ASPO. Overall, our work provides the first principled explanation for TIR's success, shifting the focus from the mere fact that tools work to why and how they enable more powerful reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。