arXiv:2412.10342cs.CVcs.AI2024-12ICCV被引 7

视觉智能体Iris通过自适应聚焦与自我优化,高效处理复杂界面任务。

Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining

  • 动态识别高信息密度区域,优先分配计算资源
  • 仅用85万标注数据实现顶尖性能,超越使用10倍数据的方法
  • 适合需要在复杂图形界面中自主操作的自动化场景

数字代理正被广泛用于网页、软件应用和操作系统等交互式数字环境中的任务自动化。虽然基于大语言模型(LLMs)的文本代理常因平台特定API需频繁更新,而依赖多模态大语言模型(MLLMs)的视觉代理能直接与图形用户界面(GUI)交互,具备更强适应性。然而,这些代理在高分辨率、视觉复杂的环境中面临显著的视觉感知挑战。本文提出Iris,一种基础视觉代理,通过两项关键创新克服这些难题:信息敏感裁剪(ISC)与自精炼双学习(SRDL)。ISC利用边缘检测算法动态识别并优先处理视觉密集区域,提升处理效率;SRDL通过双学习循环,使指代(描述UI元素)与定位(找到元素)能力相互增强,且无需额外标注数据。实证评估表明,Iris仅用85万条GUI标注即在多个基准上达到顶尖表现,优于使用10倍训练数据的方法,其改进也显著提升了网页和操作系统代理的下游任务性能。

原文摘要 · Abstract (English)

Digital agents are increasingly employed to automate tasks in interactive digital environments such as web pages, software applications, and operating systems. While text-based agents built on Large Language Models (LLMs) often require frequent updates due to platform-specific APIs, visual agents leveraging Multimodal Large Language Models (MLLMs) offer enhanced adaptability by interacting directly with Graphical User Interfaces (GUIs). However, these agents face significant challenges in visual perception, particularly when handling high-resolution, visually complex digital environments. This paper introduces Iris, a foundational visual agent that addresses these challenges through two key innovations: Information-Sensitive Cropping (ISC) and Self-Refining Dual Learning (SRDL). ISC dynamically identifies and prioritizes visually dense regions using a edge detection algorithm, enabling efficient processing by allocating more computational resources to areas with higher information density. SRDL enhances the agent's ability to handle complex tasks by leveraging a dual-learning loop, where improvements in referring (describing UI elements) reinforce grounding (locating elements) and vice versa, all without requiring additional annotated data. Empirical evaluations demonstrate that Iris achieves state-of-the-art performance across multiple benchmarks with only 850K GUI annotations, outperforming methods using 10x more training data. These improvements further translate to significant gains in both web and OS agent downstream tasks.

视觉代理GUI自动化多模态模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。