arXiv:2607.10079cs.AIcs.CL2026-07

首个统一网页操作与引导生成的多模态基准,让AI更像真人用网页。

MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation

论文配图:MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation
图 1 · 摘自论文原文
  • 构建多模态任务基准,用截图指导点击和写引导文本
  • 90亿参数模型成功率从6.9%提升至13.2%,仍不足40%完成率
  • 适合研究智能助手、自动化测试与人机交互的开发者

数字采纳平台(DAP)是嵌入网页的引导层,帮助用户快速上手陌生界面。但真实任务往往涉及跨页面状态的一系列操作,现有研究将网页自动化与引导生成视为独立问题,且多依赖文本化页面结构(如DOM树),而非人类实际操作的渲染截图。本文提出MAG,首个将任务执行与引导文本生成统一为多模态任务的基准,采用两种基于截图的定位方式:标记集合选择与原始像素坐标。我们还构建了完整工具链,涵盖大模型辅助标注、人工验证、训练、实时环境评估及联合动作与引导评分。基于此工具链,我们评估前沿API模型与开源多模态模型,并报告详细分析。最后设计了结合专家轨迹的GRPO训练方法,使90亿参数代理的成功率从6.9%提升至13.2%,同时改善引导质量。即便最强模型也仅完成不足40%的任务,表明该领域仍有巨大研究空间。

原文摘要 · Abstract (English)

Digital Adoption Platforms (DAPs) are embedded overlays widely used on web systems to guide users through operations inside a page, helping them get started with unfamiliar interfaces quickly. Completing a real task, however, rarely means clicking a few buttons on a single page: it takes a sequence of actions that unfolds across changing page states. Prior studies have also treated automated web agent actions and guide text generation as two separate problems, and most of them feed models textual page representations such as the DOM or accessibility trees rather than the rendered screens that humans actually operate on. In this work we introduce MAG, the first benchmark that unifies task execution and guide writing into a single Multimodal Action and Guide task, with two grounding schemes over screenshots: Set-of-Mark element selection and raw pixel coordinates. We further build a complete harness for this compound task, covering annotation with LLM assistance and human verification, training, evaluation in live environments, and joint metrics for actions and guides. With this harness we evaluate frontier API models and open multimodal models, and report detailed analyses. Finally, we design a GRPO training method augmented with expert trajectories, which nearly doubles the success rate of a supervised 9B agent (from 6.9% to 13.2%) and improves guide quality at the same time. Even the strongest model completes fewer than 40% of the tasks, leaving ample room for future research.

网页代理多模态自动引导基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。