让智能体从交互中学习环境动态，提升长任务成功率。
EnvRL: Learn from Environment Dynamics in Agentic Reinforcement Learning
- 通过状态预测和逆向动力学辅助训练，挖掘交互轨迹中的隐含信号。
- 在ALFWorld和WebShop上，成功率分别提升至77.4%和67.0%。
- 适合做长周期任务的LLM智能体，尤其关注环境建模的强化学习研究者。
强化学习（RL）已成为训练大语言模型（LLMs）作为智能体的强大范式。然而，传统用于长周期智能体任务的RL方法常面临稀疏结果奖励的问题。我们指出，这一问题忽略了回放轨迹中蕴含的丰富环境动态信息。交互经验本身可作为隐式监督信号，揭示环境的内在转移机制，并帮助智能体构建更准确的内部环境模型。为此，本文提出EnvRL框架，通过状态预测和逆向动力学两个辅助目标，将环境动态学习融入智能体强化学习。在主目标联合优化下，智能体可从自身交互经验中内化环境动态。在两个长周期智能体基准测试上的大量实验表明，与仅使用RL的基线相比，EnvRL显著提升了成功率：以GRPO训练时，Qwen-2.5-1.5B-Instruct在ALFWorld上从72.8%提升至77.4%，在WebShop上从56.8%提升至67.0%。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as a powerful paradigm for training Large Language Models (LLMs) as agents. However, conventional RL methods for long-horizon agentic tasks often struggle with sparse outcome rewards. Intuitively, this overlooks the rich environment dynamics information contained in rollout interaction trajectories. We argue that the interaction experience inherently serves as an implicit supervision signal, reveals the underlying transition mechanisms of the environment, and enables the agent to construct a more accurate internal model of the environment.. Therefore, in this work, we investigate how to leverage this additional signal to improve policy learning. Specifically, we propose EnvRL, a framework that incorporates environment dynamics learning into agentic RL via two auxiliary objectives: state prediction and inverse dynamics. By jointly optimizing with the primary RL objective, we encourage the agent to internalize environment dynamics from its own interaction experience. Extensive experiments on two long-horizon agentic benchmarks demonstrate that EnvRL achieves significant improvements on success-rates over RL-only baselines, e.g., when trained with GRPO, lifting Qwen-2.5-1.5B-Instruct from 72.8% to 77.4% on ALFWorld, and from 56.8% to 67.0% on WebShop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。