用模拟环境训练语言模型用工具推理，不依赖真实API也能高效准确。
Adaptive Tool Generation with Models as Tools and Reinforcement Learning
- 通过多智能体模拟生成带验证的推理过程，替代真实API调用。
- 在4个复杂问答数据集上达到与实时API系统相当的准确率。
- 适合需要稳定、低成本工具推理的应用场景，如教育或企业系统。
工具增强型语言模型展现出强大能力，但其对实时API的依赖带来了训练和部署中的可扩展性与可靠性问题。我们提出MTR——一种以模拟为核心的工具增强推理训练框架。MTR不依赖真实API，而是从包含模式验证的模拟观测结果的完整ReAct轨迹中学习。该方法采用多智能体架构：ToolMaker生成特定任务的OpenAI兼容工具接口，AutoAgent生成结构化的思考-行动-观察序列，ToolActor模拟真实响应。训练分为两阶段：第一阶段监督微调（SFT）学习‘推理语法’；第二阶段群体相对策略优化（GRPO）通过综合奖励函数优化策略，平衡答案正确性与内部一致性。在四个多跳问答基准（HotpotQA、MuSiQue、2WikiMultiHopQA、Bamboogle）上，MTR取得与实时API系统相当的精确匹配（EM）分数，尤其在推理密集型任务中表现优异，表明无需真实交互即可从结构化轨迹中学会有效的工具推理。
原文摘要 · Abstract (English)
Tool-augmented language models have demonstrated strong capabilities, but their reliance on live API access creates scalability and reliability challenges during training and deployment. We propose MTR, a simulation-first training framework for tool-augmented reasoning. Instead of relying on live APIs, MTR learns from complete ReAct traces with schema-validated, simulated observations. Our approach operates through a multi-agent architecture where a ToolMaker generates task-specific, OpenAI-compatible tool interfaces, an AutoAgent produces structured think-act-observe sequences, and a ToolActor simulates realistic responses. Training proceeds in two stages: Stage-1 Supervised Fine-Tuning (SFT) teaches 'trace grammar' from complete reasoning sequences; Stage-2 Group Relative Policy Optimization (GRPO) optimizes strategy with a composite trace reward that balances answer correctness and internal consistency. Across four multi-hop QA benchmarks (HotpotQA, MuSiQue, 2WikiMultiHopQA, Bamboogle), MTR attains competitive Exact Match (EM) scores to live-API systems and excels on reasoning-intensive tasks, suggesting that effective tool reasoning can be learned from structured traces without live interactions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。