用强化学习提升大模型调用工具能力，效果超越GPT-4o。
Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning
- 通过规则化强化学习，仅奖励工具调用格式与功能正确性。
- 7B/14B模型在多个基准上优于GPT-4o，实现更优泛化。
- 无需标注推理轨迹，适合追求自主推理的场景。
让大语言模型具备外部工具调用能力已成为扩展其功能的重要策略。以往方法多依赖强模型蒸馏轨迹进行监督微调，导致推理模式模仿化，限制泛化能力。本文提出基于规则的强化学习训练框架，构建Nemotron-Research-Tool-N1系列工具调用推理模型。该方法不强制监督中间推理轨迹，仅以二元奖励评估工具调用的格式有效性与功能正确性。轻量级监督使模型可自主发展推理策略，无需依赖标注轨迹。在多个主流基准上，Tool-N1-7B/14B显著优于GPT-4o。我们系统研究了规则化强化学习策略设计，基于5,518条蒸馏推理轨迹对比了SFT、RL及SFT-then-RL流程，发现广泛采用的SFT-then-RL范式未必优于纯强化学习。
原文摘要 · Abstract (English)
Enabling large language models with external tools has become a pivotal strategy for extending their functionality beyond text space. To enhance LLMs' tool-calling abilities, previous approaches primarily rely on supervised fine-tuning (SFT) with trajectories distilled from stronger models, often resulting in imitative reasoning that limits generalization. In this work, we explore rule-based reinforcement learning to enhance tool-calling in LLMs, resulting in Nemotron-Research-Tool-N1, a series of tool-calling reasoning models. Rather than enforcing supervision over intermediate distilled reasoning traces, Tool-N1 is trained with a binary RL reward that assesses only the format validity and functional correctness of tool invocations. This lightweight supervision allows the model to develop reasoning strategies independently, without relying on annotated trajectories. Experiments on several major benchmarks show that Tool-N1-7B/14B clearly outperform GPT-4o. We conduct a systematic study on the design of rule-based reinforcement learning strategies for training tool-calling models. Using 5,518 distilled reasoning trajectories, we compare SFT, RL, and the SFT-then-RL pipeline, finding that the widely adopted SFT-then-RL paradigm does not necessarily outperform pure RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。