让界面智能体学会主动看图,用裁剪放大等工具提升识别精度。
GUI-Eyes: Tool-Augmented Perception for Visual Grounding in GUI Agents
- 分两阶段决策是否调用视觉工具及如何使用
- 仅用3000样本在ScreenSpot-Pro上达44.8%准确率
- 适合构建高效、自适应的自动化界面助手
视觉语言模型(VLMs)与强化学习(RL)的进步推动了图形用户界面(GUI)自动化的发展。然而,现有方法多依赖静态、单次视觉输入和被动感知,缺乏根据任务动态决定何时、是否以及如何观察界面的能力。本文提出GUI-Eyes,一种用于GUI任务中主动视觉感知的强化学习框架。为获取更丰富的信息,该智能体在双阶段推理过程中学习策略性地决定是否调用视觉工具(如裁剪或缩放)。为此,我们设计了一种渐进式感知策略,将决策分解为粗粒度探索与细粒度定位,由两级策略协调。此外,我们构建了一个空间连续的奖励函数,结合位置接近度与区域重叠度,以提供密集监督,缓解GUI环境中常见的奖励稀疏问题。在ScreenSpot-Pro基准上,GUI-Eyes-3B仅使用3000个标注样本即达到44.8%的定位准确率,显著优于监督与基于强化学习的基线方法。结果表明,通过分阶段策略推理与细粒度奖励反馈实现的工具感知主动感知,对构建鲁棒且数据高效的GUI智能体至关重要。
原文摘要 · Abstract (English)
Recent advances in vision-language models (VLMs) and reinforcement learning (RL) have driven progress in GUI automation. However, most existing methods rely on static, one-shot visual inputs and passive perception, lacking the ability to adaptively determine when, whether, and how to observe the interface. We present GUI-Eyes, a reinforcement learning framework for active visual perception in GUI tasks. To acquire more informative observations, the agent learns to make strategic decisions on both whether and how to invoke visual tools, such as cropping or zooming, within a two-stage reasoning process. To support this behavior, we introduce a progressive perception strategy that decomposes decision-making into coarse exploration and fine-grained grounding, coordinated by a two-level policy. In addition, we design a spatially continuous reward function tailored to tool usage, which integrates both location proximity and region overlap to provide dense supervision and alleviate the reward sparsity common in GUI environments. On the ScreenSpot-Pro benchmark, GUI-Eyes-3B achieves 44.8% grounding accuracy using only 3k labeled samples, significantly outperforming both supervised and RL-based baselines. These results highlight that tool-aware active perception, enabled by staged policy reasoning and fine-grained reward feedback, is critical for building robust and data-efficient GUI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。