用代码生成1000个可执行的合成环境，让智能体高效训练并泛化
Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning
- 基于代码与数据库构建可执行合成环境，状态更可靠
- 在3个基准上仅用合成环境训练即实现强泛化能力
- 适合做多轮工具使用与强化学习的科研人员
大语言模型的发展使智能体能够与工具和环境进行多轮交互，但训练规模受限于多样且可靠的环境不足。本文提出代理世界模型（AWM），一个全合成环境生成管道。通过该管道，我们构建了1000个涵盖日常场景的合成环境，智能体可在其中与丰富的工具集交互并获取高质量观测。这些环境由代码驱动并依托数据库，状态转移比仅靠大语言模型模拟的环境更可靠、一致。同时，相比从真实环境收集轨迹，交互效率更高。为验证资源有效性，我们在多个任务上进行了大规模强化学习实验。得益于可执行环境与可访问的数据库状态，可设计可靠奖励函数。在三个基准测试中，仅在合成环境中训练即可实现出色的分布外泛化性能。代码已开源：https://github.com/Snowflake-Labs/agent-world-model。
原文摘要 · Abstract (English)
Recent advances in large language model (LLM) have empowered autonomous agents to perform multi-turn interactions with tools and environments. However, scaling such agent training is limited by the lack of diverse and reliable environments. In this paper, we propose Agent World Model (AWM), a fully synthetic environment generation pipeline. Using this pipeline, we scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets and obtain high-quality observations. Notably, these environments are code-driven and backed by databases, providing more reliable and consistent state transitions than environments simulated by LLMs. Moreover, they enable more efficient agent interaction compared with collecting trajectories from realistic environments. To demonstrate the effectiveness of this resource, we perform large-scale reinforcement learning for multi-turn tool-use agents. Thanks to the fully executable environments and accessible database states, we can also design reliable reward functions. Experiments on three benchmarks show that training exclusively in synthetic environments, rather than benchmark-specific ones, yields strong out-of-distribution generalization. The code is available at https://github.com/Snowflake-Labs/agent-world-model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。