让大模型先规划再调用工具，提升数学推理准确率
Implicit Hierarchical GRPO: Decoupling Tool Invocation from Execution for Tool-Integrated Mathematical Reasoning
- 分离工具调用与执行时机，先规划后执行
- 在6个数学基准上最高提升2.53%准确率
- 适合需要精确推理的数学和代码任务
大语言模型（LLMs）越来越多地通过调用工具来增强推理能力。然而，现有方法通常将工具调用与即时执行紧密耦合，这种即时交互可能破坏模型的推理连贯性并限制其表达能力，最终降低推理性能。为此，我们首次提出并形式化了在推理过程中解耦工具调用与执行的问题，引入延迟执行与显式控制以增强工具集成推理（TIR）。此外，我们提出一种分层控制框架，并理论推导出一个代理损失，使隐式分层策略能够学习到与显式分层策略等价的行为，从而得到IH-GRPO算法。在多个测试中，IH-GRPO在Qwen3-1.7B、Qwen3-4B和Qwen3-8B模型上，于六个跨域数学推理基准上分别取得1.87%、2.16%和2.53%的绝对性能提升，优于最强基线方法，且在其他领域也表现出一致的性能优势。
原文摘要 · Abstract (English)
Large language models (LLMs) have increasingly leveraged tool invocation to enhance their reasoning capabilities. However, existing approaches typically tightly couple tool invocation with immediate execution. Such immediate tool interaction may disrupt the reasoning coherence of LLMs and constrain their expressivity, ultimately degrading reasoning performance. To this end, for the first time, we propose and formalize the problem of decoupling tool invocation from execution during reasoning, and introduce delayed execution with explicit control to enhance tool-integrated reasoning (TIR). Furthermore, we propose a hierarchical control framework and theoretically derive a surrogate loss that enables an implicitly hierarchical policy to learn behavior equivalent to that of an explicit hierarchical policy, leading to the proposed IH-GRPO algorithm. Extensive experiments on IH-GRPO achieve absolute improvements of 1.87\%, 2.16\%, and 2.53\% on Qwen3-1.7B, Qwen3-4B, and Qwen3-8B across six out-of-domain mathematical reasoning benchmarks over the strongest baseline method, while also yielding consistent performance gains in other domains. Our code is available at https://github.com/Lumina04/IH-GRPO-01.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。