正确代码能显著提升大模型生成测试用例的效果。
Measuring the Influence of Incorrect Code on Test Generation
- 对比正确与错误代码，评估大模型生成测试的效果差异。
- 正确代码使测试准确率、覆盖率和缺陷检测率分别提升57%、12%和24%。
- 补充自然语言代码描述可进一步提升测试效果，适合工程实践参考。
人们普遍认为，当提示大语言模型（LLM）使用正确代码时，生成的测试用例会更准确，但这一影响的大小从未被量化。为此，我们对5个开源和6个闭源语言模型进行了全面实证研究，涵盖3个常用基准数据集及来自两个真实数据集的41个仓库级实例。结果显示，相比错误代码，使用正确代码时，LLM在测试准确率、代码覆盖率和缺陷检测率上分别提升57%、12%和24%。这些结论在真实代码中同样成立：针对错误代码生成的测试，缺陷检测率低47%。此外，提供自然语言代码描述可使准确率、覆盖率和缺陷检测率分别提升18%、4%和34%。这些发现具有直接实践意义，如通过补充描述可有效缓解测试质量下降问题。
原文摘要 · Abstract (English)
It is natural to suppose that a Large Language Model is more likely to generate correct test cases when prompted with correct code under test, compared to incorrect code under test. However, the size of this effect has never been previously measured, despite its obvious importance for both practicing software engineers and researchers. To answer the question, we conducted a comprehensive empirical study on 5 open source and 6 closed source language models, with 3 widely-used benchmark data sets together with 41 repo-level real-world examples from two different real-world data sets. Our results reveal that, when compared to incorrect code under test, LLMs prompted with correct code achieve improvements in test accuracy, code coverage, and bug detection of 57\%, 12\%, and 24\% respectively. We further show that these scientific conclusions carry over from the three benchmark data sets to the real-world code, where tests generated for incorrect code experience a 47\% worse bug detection rate. Finally, we report that improvements of +18\% in accuracy, +4\% coverage, and +34\% in bug detection can be achieved by providing natural language code descriptions. These findings have actionable conclusions. For example, the 47\% reduction in real-world bug detection is a clear concern. Fortunately, it is a concern for which our findings about the added value of descriptions offer an immediately actionable remedy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。