GTA1通过测试时扩展提升界面操作的准确性和任务完成率。
GTA1: GUI Test-time Scaling Agent
- 测试时并行采样多个操作方案,由判别模型选择最优路径。
- 在复杂高分辨率界面中实现精准点击,成功率显著提升。
- 适合需要高精度界面交互的自动化测试与智能代理场景。
图形用户界面(GUI)代理通过将用户指令逐步分解为操作建议,与动态环境中的视觉元素交互来自主完成跨平台任务(如Linux)。然而面临两大挑战:一是在庞大的动作空间中进行规划,存在多种有效方案难以抉择;二是精确对齐动作与复杂高分辨率界面中的视觉目标。本文提出测试时扩展代理GTA1,首先在每一步并行采样多个候选操作建议,由判别模型评估并选择最优方案,以计算开销换取决策质量;其次,提出一种强化学习模型,通过内在目标对齐(奖励成功点击界面元素)提升所选动作的视觉定位精度。实验表明,GTA1在接地和代理任务执行基准上均达到当前最优性能。代码与模型已开源。
原文摘要 · Abstract (English)
Graphical user interface (GUI) agents autonomously complete tasks across platforms (\eg, Linux) by sequentially decomposing user instructions into action proposals that iteratively interact with visual elements in the evolving environment. However, two main challenges arise: i) planning (\ie, the action proposal sequence) under expansive action space, where selecting an appropriate plan is non-trivial, as many valid ones may exist; ii) accurately grounding actions in complex and high-resolution interfaces, \ie, precisely interacting with visual targets. This paper investigates the aforementioned challenges with our \textbf{G}UI \textbf{T}est-time Scaling \textbf{A}gent, namely GTA1. First, we conduct test-time scaling to select the most appropriate action proposal: at each step, multiple candidate proposals are sampled and evaluated and selected by a judge model. It trades off computation for better decision quality by concurrent sampling. Second, we propose a model that improves grounding of the selected action proposals to its corresponding visual elements. Our key insight is that reinforcement learning (RL) facilitates grounding through inherent objective alignments, rewarding successful clicks on interface elements. Experimentally, GTA1 achieves state-of-the-art performance on both grounding and agent task execution benchmarks. The code and models are released here.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。