用虚拟网页世界让网页智能体‘做梦’，高效训练无需真上网。
DynaWeb: Model-Based Reinforcement Learning of Web Agents
- 构建网页世界模型，让智能体在虚拟环境中生成大量行动轨迹。
- 结合真实专家数据与虚拟推演，提升训练稳定性和效率。
- 在WebArena和WebVoyager上显著超越现有开源模型，适合高效率训练场景。
基于大语言模型（LLMs）和强化学习（RL）的自主网页智能体，是通用人工智能助手的重要进展。然而，直接在真实互联网上训练效率低、成本高且存在风险。模型基于强化学习（MBRL）通过学习环境世界模型，实现模拟交互，提供解决方案。本文提出DynaWeb，一种新型MBRL框架，通过训练一个能根据智能体动作预测自然网页表示的网页世界模型，构建合成网页环境。智能体可在此环境中‘幻想’生成海量行动轨迹，实现高效的在线强化学习。除了自由生成的策略轨迹外，DynaWeb还将真实专家轨迹随机穿插于策略轨迹中进行训练，以提升稳定性与样本效率。在WebArena和WebVoyager等挑战性基准上的实验表明，DynaWeb持续且显著提升了当前最先进开源网页智能体模型的性能。研究结果验证了通过‘想象’训练网页智能体的可行性，为规模化在线智能体强化学习提供了高效可扩展的新路径。
原文摘要 · Abstract (English)
The development of autonomous web agents, powered by Large Language Models (LLMs) and reinforcement learning (RL), represents a significant step towards general-purpose AI assistants. However, training these agents is severely hampered by the challenges of interacting with the live internet, which is inefficient, costly, and fraught with risks. Model-based reinforcement learning (MBRL) offers a promising solution by learning a world model of the environment to enable simulated interaction. This paper introduces DynaWeb, a novel MBRL framework that trains web agents through interacting with a web world model trained to predict naturalistic web page representations given agent actions. This model serves as a synthetic web environment where an agent policy can dream by generating vast quantities of rollout action trajectories for efficient online reinforcement learning. Beyond free policy rollouts, DynaWeb incorporates real expert trajectories from training data, which are randomly interleaved with on-policy rollouts during training to improve stability and sample efficiency. Experiments conducted on the challenging WebArena and WebVoyager benchmarks demonstrate that DynaWeb consistently and significantly improves the performance of state-of-the-art open-source web agent models. Our findings establish the viability of training web agents through imagination, offering a scalable and efficient way to scale up online agentic RL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。