用标准化流程评估LLM生成的Java单元测试质量
LLMs for Automated Unit Test Generation and Assessment in Java: The AgoneTest Framework
- 构建端到端评估框架，统一对比不同LLM与提示策略
- 编译通过的测试覆盖率和缺陷检测能力可媲美人工编写
- 适合测试研究者、开发者优化提示工程与模型设计
单元测试是软件开发中不可或缺但耗时的环节，用于确保代码单元正确运行。本文提出AgoneTest，一个针对大型语言模型生成（LLM）Java单元测试的自动化评估框架。该框架不旨在提出新的测试生成算法，而是为研究人员和开发者提供在真实条件下，标准化地比较不同LLM及提示策略的全流程评估方案。我们构建了Classes2Test数据集，将待测Java类与其对应的测试类进行映射，并集成突变分数、测试异味等高级评估指标，实现全面评估。实验表明，在编译通过的测试子集中，LLM生成的测试在覆盖率和缺陷检测能力上可达到或超过人工编写的测试水平。研究还发现，增强型提示策略显著提升测试质量。AgoneTest揭示了LLM在软件测试中的潜力，并为模型设计、提示工程与测试实践的未来改进提供了洞见。
原文摘要 · Abstract (English)
Unit testing is an essential but resource-intensive step in software development, ensuring individual code units function correctly. This paper introduces AgoneTest, an automated evaluation framework for Large Language Model-generated (LLM) unit tests in Java. AgoneTest does not aim to propose a novel test generation algorithm; rather, it supports researchers and developers in comparing different LLMs and prompting strategies through a standardized end-to-end evaluation pipeline under realistic conditions. We introduce the Classes2Test dataset, which maps Java classes under test to their corresponding test classes, and a framework that integrates advanced evaluation metrics, such as mutation score and test smells, for a comprehensive assessment. Experimental results show that, for the subset of tests that compile, LLM-generated tests can match or exceed human-written tests in terms of coverage and defect detection. Our findings also demonstrate that enhanced prompting strategies contribute to test quality. AgoneTest clarifies the potential of LLMs in software testing and offers insights for future improvements in model design, prompt engineering, and testing practices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。