构建统一基准评估自动GUI测试智能体，发现当前模型普遍存在短板。
GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent
- 将GUI测试拆解为意图生成、任务执行、缺陷检测三步骤，建立标准化评估框架。
- 在真实应用、注入缺陷应用和合成数据上测试，揭示主流模型在多任务中表现不佳。
- 适合研究GUI自动化、多模态大模型应用的开发者与评测人员参考。
当前GUI智能体研究集中于任务自动化，应用场景受限。本文提出GTArena——一个形式化且全面的自动化GUI测试评估环境，支持多种多模态大语言模型在统一平台上的公平比较。将测试过程分解为三个关键子任务：测试意图生成、测试任务执行与GUI缺陷检测，并基于此构建基准数据集。评估涵盖三类数据：真实移动应用、人工注入缺陷的应用及合成数据，全面检验模型在该任务中的能力。此外，提出方法分析多模态大模型在特定场景的表现与其标准基准测试能力间的相关性。实验表明，即使是最先进的模型，在所有子任务上也难以同时表现优异，凸显当前自主GUI测试能力与实际应用需求间存在显著差距，为未来研究指明方向。代码已开源：https://github.com/ZJU-ACES-ISE/ChatUITest。
原文摘要 · Abstract (English)
Nowadays, research on GUI agents is a hot topic in the AI community. However, current research focuses on GUI task automation, limiting the scope of applications in various GUI scenarios. In this paper, we propose a formalized and comprehensive environment to evaluate the entire process of automated GUI Testing (GTArena), offering a fair, standardized environment for consistent operation of diverse multimodal large language models. We divide the testing process into three key subtasks: test intention generation, test task execution, and GUI defect detection, and construct a benchmark dataset based on these to conduct a comprehensive evaluation. It evaluates the performance of different models using three data types: real mobile applications, mobile applications with artificially injected defects, and synthetic data, thoroughly assessing their capabilities in this relevant task. Additionally, we propose a method that helps researchers explore the correlation between the performance of multimodal language large models in specific scenarios and their general capabilities in standard benchmark tests. Experimental results indicate that even the most advanced models struggle to perform well across all sub-tasks of automated GUI Testing, highlighting a significant gap between the current capabilities of Autonomous GUI Testing and its practical, real-world applicability. This gap provides guidance for the future direction of GUI Agent development. Our code is available at https://github.com/ZJU-ACES-ISE/ChatUITest.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。