构建可规模化训练的网页智能体环境,显著降低资源消耗并提升执行可靠性。
WEBSERV: A Full-Stack and RL-Ready Web Environment for Training Web Agents at Scale
- 采用容器化技术与块级写时复制,实现快速启动与低存储开销。
- 支持200+并发环境,单机运行效率提升5倍以上,关键任务准确率达55.5%。
- 适配复杂网页应用,适合大规模强化学习训练,尤其适合大模型部署者。
强化学习训练网页智能体需要高效且可靠的环境。现有方案中,服务器端的Docker部署资源开销大,难以支撑大规模并行采样;浏览器端接口则存在观测噪声高、动作执行不可靠、缺乏视觉交互信号等问题。本文提出WebServ,一个全栈式、面向强化学习的网页环境。服务端采用Incus容器与块级写时复制,使环境启动延迟降低约5倍,持久化存储减少约240倍,单机支持200+独立并发环境。客户端提供基于DOM自动生成的紧凑、跨站点通用的观测与动作接口,并通过网络感知等待机制保障对现代单页应用的可靠动作执行。在WebArena-Lite上,WebServ达到当前最优单提示性能,且在GPT-4o、OpenAI-o3和Llama-3.1-8B上均优于基线。进一步在WebServ内完成Qwen3-4B与Qwen3-30B-A3B的强化学习训练,4B模型平均准确率达55.5%,超过Claude 4.5 Sonnet(50.0%)及WebAgent-R1的RL训练8B模型(51.8%)。
原文摘要 · Abstract (English)
Reinforcement learning (RL) for web agents demands environments that are both effective for evaluation and efficient enough for large-scale on-policy training. Current web environments fall short: server-side Docker setups are too resource-intensive for massive parallel rollouts, while browser-side interfaces produce noisy observations, execute actions unreliably under modern single-page applications, and omit visual interactivity cues. We introduce WebServ, a full-stack, RL-ready web environment that addresses these limitations end-to-end. On the server side, WebServ uses Incus containers with block-level copy-on-write, reducing launch latency by ~5x and persistent storage by ~240x, enabling 200+ concurrent isolated environments on a single host. On the browser side, WebServ provides a compact, site-agnostic observation and action interface derived automatically from the DOM with human-aligned interactivity cues, and a robust action execution backend using network-aware waiting for reliable SPA support. On WebArena-Lite, WebServ achieves state-of-the-art single-prompt results, with controlled comparisons confirming consistent gains across GPT-4o, OpenAI-o3, and Llama-3.1-8B over vanilla WebArena. We further train Qwen3-4B and Qwen3-30B-A3B with RL entirely within WebServ; the RL-trained 4B model achieves 55.5% mean accuracy, surpassing both Claude 4.5 Sonnet (50.0%) and the RL-trained 8B model from WebAgent-R1 (51.8%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。