用网络教程自动生成网页智能体轨迹,低成本训练高效数字助手。
AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

- 用分类模型抓取网络教程,转为结构化任务指令。
- 视觉语言模型执行指令并验证结果,生成多模态轨迹数据。
- 每条高质量轨迹成本仅0.55美元,适合大规模训练智能体。
图形用户界面(GUI)智能体可自动化跨数字环境的复杂任务,但其开发受限于高质量轨迹数据的稀缺。现有方法依赖昂贵的人工标注,难以规模化。我们提出AgentTrek,一个可扩展的数据合成管道,通过利用公开的网络教程生成网页智能体轨迹。该三阶段方法:(1) 使用专用分类模型自动采集并过滤互联网上的教程类文本;(2) 将这些文本转化为带步骤说明的结构化任务规范;(3) 采用视觉语言模型(VLM)智能体在真实环境中执行指令,同时由基于VLM的评估器验证轨迹正确性。合成轨迹涵盖多种模态,包括基于文本的HTML观测与函数调用API操作,以及基于视觉的截图观测与像素级操作。该多模态数据结合思维链推理,使智能体在文本型网页浏览基准(如WebArena)和视觉网页定位与浏览基准(如ScreenSpot Web与Multimodal Mind2Web)上均达到当前最优性能。此外,我们的全自动化方法显著降低数据收集成本,每条高质量轨迹成本仅为0.55美元,无需人工标注。本工作证明,借助网络教程进行引导重放是训练先进GUI智能体的一种可行且可扩展的策略,为更强大、自主的数字助理铺平道路。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) agents can automate complex tasks across digital environments, but their development is hindered by the scarcity of high-quality trajectory data for training. Existing approaches rely on expensive human annotation, making them unsustainable at scale. We propose AgentTrek, a scalable data synthesis pipeline that generates web agent trajectories by leveraging publicly available tutorials. Our three-stage method: (1) automatically harvests and filters tutorial-like texts from the internet using a specialized classification model, (2) transforms these texts into structured task specifications with step-by-step instructions, and (3) employs a visual-language model (VLM) agent to execute these instructions in real environments, while a VLM-based evaluator verifies trajectory correctness. The synthesized trajectories encompass multiple modalities, including text-based HTML observations with function-calling API actions, and vision-based screenshot observations with pixel-level actions. This multimodal data, enriched with chain-of-thought reasoning, enables agents to achieve state-of-the-art performance on both textual web browsing benchmarks (e.g., WebArena) and visual web grounding and browsing benchmarks (e.g., ScreenSpot Web and Multimodal Mind2Web). Furthermore, our fully automated approach significantly reduces data collection costs, achieving a cost of just $0.55 per high-quality trajectory without human annotators. Our work demonstrates that guided replay using web tutorials is a practical and scalable strategy for training advanced GUI agents, paving the way for more capable and autonomous digital assistants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。