提升多步视觉网页智能体的强化学习效率，速度最高快2.9倍。
AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agents
- 异步系统设计叠加滚动池与轻量截图处理，减少GPU空闲。
- 用常数替换轨迹归一化因子，使失败轨迹变短且成功率提升。
- 适合追求高效训练的网页智能体研究者和工业应用开发者。
以多步强化学习训练视觉-语言网页智能体计算开销大，主要源于同步模式下的GPU闲置及轨迹步数与令牌数冗余。本文提出AsyncWebRL，系统层面采用异步设计，实现滚动、梯度更新与策略刷新的重叠，并引入永恒滚动池与轻量截图处理，相比此前最快的开源同步管道WebGym，端到端训练吞吐量最高提升2.9倍。算法层面，发现多步GRPO中的轨迹归一化项1/|τ_i|是效率低下的根源：因失败轨迹普遍长于成功轨迹，该归一化会弱化失败令牌的负梯度，导致策略持续生成冗长记忆结构。改用常数1/k替代后，可压缩轨迹长度同时保持整体成功率。该方法在WebGym分布外测试集上达到新开源基准，相对此前最佳42.9%提升5.8%，其中中等难度子集提升42%，困难子集提升48%。
原文摘要 · Abstract (English)
Training vision-language web agents with multi-step RL is compute-intensive, with two dominant forms of inefficiency: idle GPUs in synchronous RL, and trajectories that use more steps and tokens than necessary. We present AsyncWebRL, which addresses both. On the system side, an asynchronous design overlaps rollout, gradient update, and policy refresh across iterations, paired with two web-agent-specific adaptations, namely an everlasting rollout pool and lightweight screenshot handling, that together deliver up to a $2.9\times$ end-to-end training-throughput speedup over the previously fastest open synchronous pipeline (WebGym). On the algorithmic side, we identify the per-trajectory normalizer $1/|τ_i|$ in multi-step GRPO as the root cause of trajectory-level and token-level inefficiency: because failures are systematically longer than successes, it down-weights the negative gradient on failed tokens, so the policy keeps producing verbose memory schemas. Replacing $1/|τ_i|$ with a constant $1/k$ breaks this coupling, contracting trajectories while preserving aggregate success. Together, these contributions set a new open-source state of the art on the WebGym out-of-distribution test split (+5.8% relative over the 42.9% prior best), with the largest gains on the harder slices (+42% relative on Medium, +48% relative on Hard).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。