用大模型和测试语言自动生成API测试,提升覆盖率与效率
Combining TSL and LLM to Automate REST API Testing: A Comparative Study
- 结合TSL与LLM,自动从OpenAPI生成测试用例
- Claude 3.5 Sonnet在成功率、覆盖率等指标上表现最优
- 适合开发团队快速构建高覆盖的API自动化测试
REST API测试的有效执行对开发团队仍是重大挑战,源于分布式系统的固有复杂性、可能场景繁多以及测试设计时间有限。穷尽所有输入组合测试不现实,常导致缺陷未被发现、人工成本高且覆盖率低。为此,我们提出RestTSLLM,利用测试规范语言(TSL)与大语言模型(LLMs)自动为REST API生成测试用例,重点解决测试场景设计与输入数据定义问题。该方案整合提示工程与自动化流水线,评估多个LLM基于OpenAPI规范生成测试的能力,衡量指标包括成功率、测试覆盖率与变异分数。结果表明,表现最佳的模型——Claude 3.5 Sonnet(Anthropic)、Deepseek R1(Deepseek)、Qwen 2.5 32b(Alibaba)和Sabia 3(Maritaca)——均能生成稳健且语境一致的API测试。其中,Claude 3.5 Sonnet在所有指标上均领先,是本研究中最适合该任务的模型。这些发现凸显了LLMs在基于API规格自动生成测试方面的潜力。
原文摘要 · Abstract (English)
The effective execution of tests for REST APIs remains a considerable challenge for development teams, driven by the inherent complexity of distributed systems, the multitude of possible scenarios, and the limited time available for test design. Exhaustive testing of all input combinations is impractical, often resulting in undetected failures, high manual effort, and limited test coverage. To address these issues, we introduce RestTSLLM, an approach that uses Test Specification Language (TSL) in conjunction with Large Language Models (LLMs) to automate the generation of test cases for REST APIs. The approach targets two core challenges: the creation of test scenarios and the definition of appropriate input data. The proposed solution integrates prompt engineering techniques with an automated pipeline to evaluate various LLMs on their ability to generate tests from OpenAPI specifications. The evaluation focused on metrics such as success rate, test coverage, and mutation score, enabling a systematic comparison of model performance. The results indicate that the best-performing LLMs - Claude 3.5 Sonnet (Anthropic), Deepseek R1 (Deepseek), Qwen 2.5 32b (Alibaba), and Sabia 3 (Maritaca) - consistently produced robust and contextually coherent REST API tests. Among them, Claude 3.5 Sonnet outperformed all other models across every metric, emerging in this study as the most suitable model for this task. These findings highlight the potential of LLMs to automate the generation of tests based on API specifications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。