用小模型生成真实测试用例,发现接口参数依赖关系。
LlamaRestTest: Effective REST API Testing with Small Language Models
- 用微调+量化的小模型分析接口文档,生成合理测试数据。
- 在12个真实服务上表现超越大模型和现有测试工具,覆盖更全。
- 适合想高效测试接口的开发者,尤其关注参数联动问题。
现代网络服务高度依赖基于OpenAPI规范的REST API。尽管大型语言模型(LLM)在测试生成方面展现出潜力,但其在REST API测试中的应用仍较少。本文提出LlamaRestTest,通过微调并量化Llama3-8B模型,利用挖掘的API示例值与参数依赖数据,生成真实测试输入,并通过分析服务器响应揭示参数间依赖关系。我们在12个真实服务(包括Spotify)上评估该方法,对比RESTGPT、RESTler、MoRest、EvoMaster及ARAT-RL等先进工具。结果表明,微调后的小模型在检测可操作的参数依赖规则和生成有效输入方面优于更大模型。研究还考察了从基础Llama3-8B到微调版本,以及2/4/8比特量化等多种配置。结果显示,小模型在测试效果与效率之间取得更好平衡,且在代码覆盖率和内部错误发现上优于现有工具,即使这些工具使用了RESTGPT增强的规格说明。
原文摘要 · Abstract (English)
Modern web services rely heavily on REST APIs, typically documented using the OpenAPI specification. The widespread adoption of this standard has resulted in the development of many black-box testing tools that generate tests based on OpenAPI specifications. Although Large Language Models (LLMs) have shown promising test-generation abilities, their application to REST API testing remains mostly unexplored. We present LlamaRestTest, a novel approach that employs two custom LLMs-created by fine-tuning and quantizing the Llama3-8B model using mined datasets of REST API example values and inter-parameter dependencies-to generate realistic test inputs and uncover inter-parameter dependencies during the testing process by analyzing server responses. We evaluated LlamaRestTest on 12 real-world services (including popular services such as Spotify), comparing it against RESTGPT, a GPT-powered specification-enhancement tool, as well as several state-of-the-art REST API testing tools, including RESTler, MoRest, EvoMaster, and ARAT-RL. Our results demonstrate that fine-tuning enables smaller models to outperform much larger models in detecting actionable parameter-dependency rules and generating valid inputs for REST API testing. We also evaluated different tool configurations, ranging from the base Llama3-8B model to fine-tuned versions, and explored multiple quantization techniques, including 2-bit, 4-bit, and 8-bit integer formats. Our study shows that small language models can perform as well as, or better than, large language models in REST API testing, balancing effectiveness and efficiency. Furthermore, LlamaRestTest outperforms state-of-the-art REST API testing tools in code coverage achieved and internal server errors identified, even when those tools use RESTGPT-enhanced specifications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。