用大模型直接执行自然语言测试用例，提升GUI测试的自动化与一致性。
On the Soundness and Consistency of LLM Agents for Executing Test Cases Written in Natural Language
- 设计带防护机制的专用代理，动态验证每步执行正确性。
- 70B参数模型在3sigma以上保持高执行一致性，接近工业标准。
- 提出弱不严谨性定义，为实际应用提供可接受的容错边界。
使用自然语言（NL）测试用例验证图形用户界面（GUI）应用正成为替代人工编写可执行测试脚本的有前景方向，后者开发成本高且难以维护。大语言模型（LLMs）的进展使直接由LLM代理执行NL测试用例成为可能。本文研究该方向，关注NL测试用例的不严谨性及执行一致性问题。由于指令模糊或代理行为不可预测，NL测试用例本身具有固有不严谨性，可能导致误报失败；重复执行同一用例也可能导致结果不一致，影响测试可靠性。为此，我们提出一种带有防护机制的算法，结合专用代理动态验证每一步执行正确性。引入评估指标衡量LLM在测试执行中的能力，并提出一个量化执行一致性的指标。我们提出‘弱不严谨性’的定义，以六西格玛工业质量标准为基准，界定可接受的执行上下文。在8个公开可用的LLM（参数量从3B到70B）上进行实验评估，结果表明当前LLM代理在GUI测试中兼具潜力与局限。实验显示，Meta Llama 3.1 70B在执行一致性方面表现良好（高于3sigma水平），具备可接受的执行能力。我们提供了原型工具、测试集与实验结果。
原文摘要 · Abstract (English)
The use of natural language (NL) test cases for validating graphical user interface (GUI) applications is emerging as a promising direction to manually written executable test scripts, which are costly to develop and difficult to maintain. Recent advances in large language models (LLMs) have opened the possibility of the direct execution of NL test cases by LLM agents. This paper investigates this direction, focusing on the impact on NL test case unsoundness and on test case execution consistency. NL test cases are inherently unsound, as they may yield false failures due to ambiguous instructions or unpredictable agent behaviour. Furthermore, repeated executions of the same NL test case may lead to inconsistent outcomes, undermining test reliability. To address these challenges, we propose an algorithm for executing NL test cases with guardrail mechanisms and specialised agents that dynamically verify the correct execution of each test step. We introduce measures to evaluate the capabilities of LLMs in test execution and one measure to quantify execution consistency. We propose a definition of weak unsoundness to characterise contexts in which NL test case execution remains acceptable, with respect to the industrial quality levels Six Sigma. Our experimental evaluation with eight publicly available LLMs, ranging from 3B to 70B parameters, demonstrates both the potential and current limitations of current LLM agents for GUI testing. Our experiments show that Meta Llama 3.1 70B demonstrates acceptable capabilities in NL test case execution with high execution consistency (above the level 3-sigma). We provide prototype tools, test suites, and results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。