arXiv:2509.17325cs.LGcs.AI2025-09中稿 · ICLR被引 11

用合成代码环境训练通用工具使用智能体,提升跨场景泛化能力。

Generalizable End-to-End Tool-Use RL with Synthetic CodeGym

  • 构建可生成多样化交互式任务的代码训练框架CodeGym。
  • Qwen2.5-32B模型在OOD测试集上准确率提升8.7点。
  • 适合研究通用智能体训练与真实工作流对齐的开发者。

工具增强型大语言模型(即LLM智能体)通过调用外部工具解决多样任务并对接现实世界。然而,现有训练方法主要依赖静态轨迹的监督微调或窄域强化学习,泛化能力差,面对新工具和未知流程时易失效。由于代码执行体现真实工作流的结构模式,我们以编程问题为结构化基础,构建包含多种任务配置的工具使用智能体训练环境。为此提出CodeGym,一个可扩展的框架,能合成多样、可验证、可控制的多轮工具使用环境,支持智能体主动探索与掌握各类工作流。CodeGym通过提取原子函数或逻辑为可调用工具,将静态编程题转化为可验证任务,覆盖广泛工具执行流程。不同规模与思维链配置的模型在CodeGym中训练后表现出一致的分布外泛化能力;例如,Qwen2.5-32B-Instruct在OOD基准τ-Bench上绝对准确率提升8.7点。结果表明,CodeGym是迈向可扩展通用强化学习环境的重要一步,有助于训练符合真实世界智能体工作流的行为。

原文摘要 · Abstract (English)

Tool-augmented large language models (LLMs), hereafter LLM agents, leverage external tools to solve diverse tasks and interface with the real world. However, current training practices largely rely on supervised fine-tuning (SFT) over static trajectories or reinforcement learning (RL) on narrow tasks, which generalize poorly beyond development settings and lead to brittleness with new tools and unseen workflows. Because code execution reflects many structural patterns of real-world workflows, we use coding problems as a structured substrate to build tool-use agent training environments with diverse task configurations. To this end, we introduce CodeGym, a scalable framework that synthesizes diverse, verifiable, and controllable multi-turn tool-use environments for agent RL, enabling LLM agents to explore and master various workflows actively. CodeGym converts static coding problems into interactive environments by extracting atomic functions or logic into callable tools, yielding verifiable tasks that span various tool-execution workflows. Models of varying sizes and chain-of-thought configurations trained in CodeGym exhibit consistent out-of-distribution generalizability; for example, Qwen2.5-32B-Instruct achieves an absolute accuracy gain of 8.7 points on the OOD benchmark $τ$-Bench. These results highlight CodeGym as a step toward scalable general-purpose RL environments for training tool-use behaviors that align with real-world agent workflows.

工具使用强化学习代码生成泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。