arXiv:2608.06197cs.AI2026-08被引 1

用内部回放替代外部环境,让大模型智能体自己模拟环境反应。

EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning

论文配图:EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning
图 1 · 摘自论文原文
  • 让智能体交替扮演行动者和环境角色,通过回放学习动作与响应的关系。
  • 在多个基准上超越依赖外部环境的基线,且跨模型规模表现稳定提升。
  • 推理时可私有回放,少量预算内不需外部交互即获额外收益,适合资源受限场景。

长周期工具使用的大语言模型智能体训练通常依赖真实或合成的可执行环境,其构建与验证成本高;或依赖难以对齐的外部模拟器。本文提出EnvACE,一种基于世界回放的智能体强化学习方法,将训练中的外部环境交互替换为内部回放。策略在行动与回放间交替:先生成工具调用,再扮演环境生成该动作引发的响应,并基于回放结果决策。两个角色联合端到端优化,使用任务成功奖励。通过世界回放,策略将其参数中内化了动作与环境响应的关系,形成直接支持决策的内部世界模型。在BFCL-v4、tau^2-Bench、VitaBench和FinMCP-Bench上,EnvACE实现强且可迁移的性能,整体评估优于环境扩展基线。控制实验进一步表明,世界回放在不同模型规模下均持续提升策略学习。测试时,内化世界模型可在正式执行前进行私有回放,在适度回放预算下无需额外外部交互即获得增益。研究确立世界回放作为突破外部环境限制的新型智能体训练路径。代码已公开于https://github.com/Within-yao/EnvACE。

原文摘要 · Abstract (English)

Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.

强化学习大模型智能体世界模型内部回放

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。