让大模型更稳定地用工具完成多轮任务，效果超更大模型。
RLFactory: A Plug-and-Play Reinforcement Learning Post-Training Framework for LLM Multi-Turn Tool-Use
- 异步调用+解耦架构，解决工具接口不一问题。
- 在NQ数据集上达0.486得分，训练速度提升6.8倍。
- 适合想快速增强模型工具使用能力的研究者。
大型语言模型在基础推理上表现优异，但在需要与外部工具交互的任务上仍显不足。本文提出RLFactory，一个即插即用的强化学习后训练框架，用于多轮工具使用。该框架通过异步非阻塞调用器和解耦的工具/训练架构，解决了工具异构性与接口问题带来的调用不稳定与适应性差；通过支持规则、模型判断与工具验证信号的奖励层，满足多样评估需求。通过引入工具反馈的观察标记重构马尔可夫决策过程，实现模型、工具与环境间的闭环反馈，并采用生成-解析-调用-更新的工作流实现动态策略优化。在Search-R1数据集上，基于Qwen3-4B模型，RLFactory在Natural Questions（NQ）数据集上取得0.486的测试分数，优于采用类似方法训练的更大模型（如Qwen2.5-7B-Instruct-GRPO的0.473），同时训练吞吐量提升6.8倍。RLFactory为真实场景中增强大模型的多轮工具使用能力提供了低门槛、高适应性的解决方案。代码地址：https://github.com/Simple-Efficient/RL-Factory。
原文摘要 · Abstract (English)
Large language models excel at basic reasoning but struggle with tasks that require interaction with external tools. We present RLFactory, a plug-and-play reinforcement learning post-training framework for multi-round tool use. RLFactory tackles (i) tool-call stability and adaptability amid tool heterogeneity and interface issues via an asyncio-based asynchronous caller and a decoupled tool/training architecture, and (ii) diverse evaluation needs via a reward layer supporting rule-based, model-judgment, and tool-verification signals. It reconstructs the MDP by introducing observation markers from tool feedback, closing the loop among model, tools, and environment, and implements a generate-parse-invoke-update workflow for dynamic policy optimization. On Search-R1 with Qwen3-4B, RLFactory achieves a 0.486 test score on the Natural Questions (NQ) dataset, surpassing larger models trained with similar techniques (e.g., Qwen2.5-7B-Instruct-GRPO at 0.473), and increases training throughput by 6.8x. RLFactory provides a low-barrier, highly adaptable framework for strengthening multi-round tool use of LLMs in real-world scenarios. Code: https://github.com/Simple-Efficient/RL-Factory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。