arXiv:2509.01055cs.AIcs.CL2025-09被引 83

构建统一工具框架,让大模型在多轮任务中高效调用代码、搜索等工具。

VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use

  • 设计模块化框架,支持多种工具通过标准接口接入。
  • 异步执行使训练速度提升近2倍,突破同步瓶颈。
  • 适配数学推理、编程、搜索等6类任务,开箱即用。

基于可验证奖励的强化学习(RLVR)在提升大模型推理能力方面表现优异,但局限于单轮交互且缺乏工具集成。现有具身强化学习带工具使用(ARLT)方法虽能处理多轮交互,但依赖特定代码库,存在碎片化、同步执行瓶颈及跨领域扩展性差等问题,阻碍了社区推广与算法创新。本文提出VerlTool,一个统一且模块化的框架,通过系统化设计解决上述问题。其四大贡献为:(1) 与VeRL上游对齐,保障兼容性并简化维护;(2) 统一工具管理,通过标准化API支持代码执行、搜索、SQL数据库、视觉处理等多种模态;(3) 异步回放执行,消除同步瓶颈,实现近2倍速度提升;(4) 在6个ARLT领域进行全面评估,性能媲美专用系统。该框架将ARLT形式化为包含多模态观测(文本/图像/视频)的多轮轨迹,超越传统单轮RLVR范式。我们在数学推理、知识问答、SQL生成、视觉推理、网页搜索和软件工程任务上训练与评估模型,结果与专用系统相当,同时提供统一训练基础设施。模块化插件架构仅需轻量级Python定义即可快速集成新工具,显著降低开发成本,为工具增强型强化学习研究提供可扩展基础。代码已开源:https://github.com/TIGER-AI-Lab/verl-tool。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated success in enhancing LLM reasoning capabilities, but remains limited to single-turn interactions without tool integration. While recent Agentic Reinforcement Learning with Tool use (ARLT) approaches have emerged to address multi-turn tool interactions, existing works develop task-specific codebases that suffer from fragmentation, synchronous execution bottlenecks, and limited extensibility across domains. These inefficiencies hinder broader community adoption and algorithmic innovation. We introduce VerlTool, a unified and modular framework that addresses these limitations through systematic design principles. VerlTool provides four key contributions: (1) upstream alignment with VeRL ensuring compatibility and simplified maintenance, (2) unified tool management via standardized APIs supporting diverse modalities including code execution, search, SQL databases, and vision processing, (3) asynchronous rollout execution achieving near 2$\times$ speedup by eliminating synchronization bottlenecks, and (4) comprehensive evaluation demonstrating competitive performance across 6 ARLT domains. Our framework formalizes ARLT as multi-turn trajectories with multi-modal observation tokens (text/image/video), extending beyond single-turn RLVR paradigms. We train and evaluate models on mathematical reasoning, knowledge QA, SQL generation, visual reasoning, web search, and software engineering tasks, achieving results comparable to specialized systems while providing unified training infrastructure. The modular plugin architecture enables rapid tool integration requiring only lightweight Python definitions, significantly reducing development overhead and providing a scalable foundation for tool-augmented RL research. Our code is open-sourced at https://github.com/TIGER-AI-Lab/verl-tool.

强化学习工具使用大模型框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。