arXiv:2605.29532cs.SEcs.AI2026-05

构建开放测试场景,让AI自动发现安卓应用界面缺陷。

GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing

论文配图:GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing
图 1 · 摘自论文原文
  • 设计新基准GUITestScape,覆盖61个真实安卓应用与508种缺陷。
  • 提出GUIJudge评估器,可分解分析AI测试过程中的检测能力。
  • 适合研究自动化测试、MML模型评估的开发者和研究人员。

探索式GUI测试对多模态大模型代理(MLLM)极具挑战:在无预设测试脚本的情况下,代理需自主导航应用并发现缺陷。然而,当前评估存在两方面不足:其一,现有基准几乎只关注交互缺陷,忽略显示类缺陷;其二,评估协议依赖预定义缺陷标注,将测试过程简化为单一终点判断,混淆了本质不同的故障模式。为此,本文提出GUITestScape,一个包含61个真实世界Android应用和508个预设缺陷(涵盖交互与显示类型)的交互式基准,并引入GUIJudge——一种开放集评估器,能将代理的测试轨迹分解为可独立诊断的能力模块。实验表明,GUIJudge实现了超越预定义标注的可靠过程感知评估,显著优于所有基线方法。在GUITestScape上的基准测试进一步揭示,现有模型在两类缺陷上的检测仍是主要瓶颈;将GUIJudge的验证模块集成至现有代理,无需重训练即可显著提升其检测性能。

原文摘要 · Abstract (English)

Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an application and discover defects through its own interaction. However, current evaluation falls short on two fronts. First, existing benchmarks focus almost exclusively on interaction defects, leaving display defects outside the evaluation frame. Second, evaluation protocols are bound to predefined defect annotations, collapsing the testing process into a single end-state judgment that conflates qualitatively distinct failure modes. To address these challenges, we present GUITestScape, an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types, and introduce GUIJudge, an open-set evaluator that decomposes an agent's testing trajectory into independently diagnosable capabilities. Experimental results demonstrate that GUIJudge achieves reliable process-aware evaluation beyond predefined annotations, substantially outperforming all baselines. Benchmarking on GUITestScape further reveals that detection remains the critical bottleneck for existing models across both defect types, and that integrating GUIJudge's verifiers into existing agents significantly boosts their detection performance without retraining.

GUI测试MML模型缺陷检测评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。