arXiv:2603.26266cs.AIcs.CV2026-03中稿 · ECCV被引 2

用网页教学视频自动补全GUI智能体的领域知识,解决其操作不熟的问题。

GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation

  • 通过字幕分析与三阶段检索,从网络视频中精准找出手册类教程。
  • 全自动标注生成任务规划与界面定位知识,提升智能体在特定软件中的表现。
  • 无需修改模型即可应用,适合各类GUI智能体快速增强领域适应性。

大型视觉语言模型赋予了GUI智能体强大的界面理解与交互能力,但由于训练时缺乏特定领域软件操作数据,这些智能体存在显著的领域偏差——对特定应用的操作流程(规划)和界面元素布局(定位)不熟悉,限制了其实际任务表现。本文提出GUIDE(GUI去偏:基于教学视频驱动的专家知识),一种无需训练、即插即用的框架,通过检索-增强的自动化标注流水线,自主从网络教学视频中获取领域专属知识。GUIDE引入两项核心创新:第一,基于字幕的Video-RAG流水线,通过三级渐进式检索(领域分类、主题提取、相关性匹配)识别任务相关的教学视频;第二,基于逆动力学范式的全自动标注流程,将连续关键帧与检测出的UI元素输入视觉语言模型,推断出所需的任务规划与界面定位知识,并注入智能体对应模块,同时解决两种领域偏差。在OSWorld上的大量实验表明,GUIDE作为即插即用组件,适用于多智能体系统与单模型智能体,均带来超过5%的性能提升并减少执行步骤,且不需修改任何模型参数或结构,验证了其作为架构无关增强手段的有效性。

原文摘要 · Abstract (English)

Large vision-language models have endowed GUI agents with strong general capabilities for interface understanding and interaction. However, due to insufficient exposure to domain-specific software operation data during training, these agents exhibit significant domain bias - they lack familiarity with the specific operation workflows (planning) and UI element layouts (grounding) of particular applications, limiting their real-world task performance. In this paper, we present GUIDE (GUI Unbiasing via Instructional-Video Driven Expertise), a training-free, plug-and-play framework that resolves GUI agent domain bias by autonomously acquiring domain-specific expertise from web tutorial videos through a retrieval-augmented automated annotation pipeline. GUIDE introduces two key innovations. First, a subtitle-driven Video-RAG pipeline unlocks video semantics through subtitle analysis, performing progressive three-stage retrieval - domain classification, topic extraction, and relevance matching - to identify task-relevant tutorial videos. Second, a fully automated annotation pipeline built on an inverse dynamics paradigm feeds consecutive keyframes enhanced with UI element detection into VLMs, inferring the required planning and grounding knowledge that are injected into the agent's corresponding modules to address both manifestations of domain bias. Extensive experiments on OSWorld demonstrate GUIDE's generality as a plug-and-play component for both multi-agent systems and single-model agents. It consistently yields over 5% improvements and reduces execution steps - without modifying any model parameters or architecture - validating GUIDE as an architecture-agnostic enhancement to bridge GUI agent domain bias.

GUI智能体视频检索领域偏差自动化标注

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。