arXiv:2605.06326cs.CL2026-05被引 1

让大模型学会用工具推理,同时不丢掉纯文本思考能力。

Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning

  • 通过精选适合用工具解的题目来训练模型使用工具
  • 控制工具调用比例,防止模型忘记纯文本推理能力
  • 先用奖励验证的强化学习优化,提升工具使用效果

工具集成推理(TIR)为突破纯文本推理的局限提供了直接路径。然而我们发现,即使强模型几乎不真正调用工具,仅引入工具评估反而会降低推理表现。本文研究如何在不牺牲纯文本推理能力的前提下,让强模型自然学会使用工具,并提出一套完整的TIR训练方案。关键发现包括:(i) 监督微调的效果依赖于教师轨迹的可学性,应优先选择天然适合工具辅助的问题;(ii) 控制工具使用轨迹的比例可缓解对纯文本推理能力的灾难性遗忘;(iii) 以 pass@k 和响应长度为目标优化,而非训练损失,能最大化TIR微调收益并保留强化学习探索空间;(iv) 基于良好微调初始化和防模式坍缩机制的稳定强化学习(RLVR)阶段,提供简单而高效解决方案。将该方案应用于4B和30B规模的Qwen3模型,在多项开源基准上达到领先水平,例如在AIME 2025测试中分别取得96.7%和99.2%的准确率。

原文摘要 · Abstract (English)

Tool-integrated reasoning (TIR) offers a direct way to extend thinking models beyond the limits of text-only reasoning. Paradoxically, we observe that tool-enabled evaluation can degrade reasoning performance even when the strong thinking models make almost no actual tool calls. In this paper, we investigate how to inject natural tool-use behavior into a strong thinking model without sacrificing its no-tool reasoning ability, and present a comprehensive TIR recipe. We highlight that (i) the effectiveness of TIR supervised fine-tuning (SFT) hinges on the learnability of teacher trajectories, which should prioritize problems inherently suited for tool-augmented solutions; (ii) controlling the proportion of tool-use trajectories could mitigate the catastrophic forgetting of text-only reasoning capacity; (iii) optimizing for pass@k and response length instead of training loss could maximize TIR SFT gains while preserving headroom for reinforcement learning (RL) exploration; (iv) a stable RL with verifiable rewards (RLVR) stage, built upon suitable SFT initialization and explicit safeguards against mode collapse, provides a simple yet remarkably effective solution. When applied to Qwen3 thinking models at 4B and 30B scales, our recipe yields models that achieve state-of-the-art performance in a wide range of benchmarks among open-source models, such as 96.7% and 99.2% on AIME 2025 for 4B and 30B, respectively.

工具推理大模型训练强化学习Qwen

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。