提出可自适应网页导航的智能体,实测成功率超70%。
OpAgent: Operator Agent for Web Navigation
- 分层多任务微调构建指令理解能力,支持网页操作
- 在线强化学习结合混合奖励机制,长序列导航成功率提升至38.1%
- 模块化智能体框架实现自我纠错,突破当前最优71.6%成功率
为完成用户指令,自主网页代理需应对真实网站固有的复杂性与动态性。传统方法主要依赖监督微调或离线强化学习,使用静态数据集,但受限于分布偏移问题,无法捕捉开放网络中随机状态转移与实时反馈。本文提出一种鲁棒的在线强化学习网页代理,通过与开放网页直接、迭代交互优化策略。核心创新包括:1)分层多任务微调,整合规划、执行与定位三类功能数据集,训练具备强指令跟随能力的视觉语言模型(VLM);2)野外在线代理强化学习,构建专用交互环境与强化学习流程,引入无真值依赖的WebJudge综合评估与基于规则的决策树(RDT)进度奖励,有效缓解长程导航中的信用分配难题;该系统在WebArena上达到38.1%的通过率(pass@5),超越所有现有单体基线。3)提出模块化代理框架OpAgent,集成规划器、定位器、反思器与总结器,实现稳健容错与自我修正,性能达新最先进水平,成功率达到71.6%。
原文摘要 · Abstract (English)
To fulfill user instructions, autonomous web agents must contend with the inherent complexity and volatile nature of real-world websites. Conventional paradigms predominantly rely on Supervised Fine-Tuning (SFT) or Offline Reinforcement Learning (RL) using static datasets. However, these methods suffer from severe distributional shifts, as offline trajectories fail to capture the stochastic state transitions and real-time feedback of unconstrained wide web environments. In this paper, we propose a robust Online Reinforcement Learning WebAgent, designed to optimize its policy through direct, iterative interactions with unconstrained wide websites. Our approach comprises three core innovations: 1) Hierarchical Multi-Task Fine-tuning: We curate a comprehensive mixture of datasets categorized by functional primitives -- Planning, Acting, and Grounding -- establishing a Vision-Language Model (VLM) with strong instruction-following capabilities for Web GUI tasks. 2) Online Agentic RL in the Wild: We develop an online interaction environment and fine-tune the VLM using a specialized RL pipeline. We introduce a Hybrid Reward Mechanism that combines a ground-truth-agnostic WebJudge for holistic outcome assessment with a Rule-based Decision Tree (RDT) for progress reward. This system effectively mitigates the credit assignment challenge in long-horizon navigation. Notably, our RL-enhanced model achieves a 38.1\% success rate (pass@5) on WebArena, outperforming all existing monolithic baselines. 3) Operator Agent: We introduce a modular agentic framework, namely \textbf{OpAgent}, orchestrating a Planner, Grounder, Reflector, and Summarizer. This synergy enables robust error recovery and self-correction, elevating the agent's performance to a new State-of-the-Art (SOTA) success rate of \textbf{71.6\%}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。