提出GBCV框架,系统评估大模型生成测试用例的能力
A Systematic Approach for Assessing Large Language Models' Test Case Generation Capability
- 基于控制流与变量使用结构,自动构建从简单到复杂的测试程序
- GPT-4o在复杂结构中表现更优,但所有模型在算术计算上仍存缺陷
- 为大模型测试生成提供可复现的评估基准,适合测试研发人员参考
软件测试保障软件质量,但手动编写测试用例耗时费力。随着大语言模型(LLMs)兴起,利用其生成单元测试成为热点。然而,缺乏覆盖多样编程场景的标准评估基准,制约了对LLM生成测试用例能力的有效评估。为此,本文提出生成基准方法GBCV(基于控制流结构与变量使用组合的生成基准),系统构建用于评估LLM测试生成能力的程序集合。该方法融合基础控制流结构与变量使用模式,灵活生成涵盖简单至复杂程度的程序。鉴于GPT-4o和GPT-3-Turbo为公开可用模型,为模拟真实用户场景,我们采用GBCV评估其性能。结果表明:GPT-4o在复杂程序结构中表现更优;所有模型在简单条件下的边界值检测有效,但在算术计算任务中面临挑战。本研究揭示了大模型在测试生成中的优势与局限,提供了可复用的评估框架,并指明未来改进方向。
原文摘要 · Abstract (English)
Software testing ensures the quality and reliability of software products, but manual test case creation is labor-intensive. With the rise of large language models (LLMs), there is growing interest in unit test creation with LLMs. However, effective assessment of LLM-generated test cases is limited by the lack of standardized benchmarks that comprehensively cover diverse programming scenarios. To address the assessment of LLM's test case generation ability and lacking dataset for evaluation, we propose the Generated Benchmark from Control-Flow Structure and Variable Usage Composition (GBCV) approach, which systematically generates programs used for evaluating LLMs' test generation capabilities. By leveraging basic control-flow structures and variable usage, GBCV provides a flexible framework to create a spectrum of programs ranging from simple to complex. Because GPT-4o and GPT-3-Turbo are publicly accessible models, to present real-world regular user's use case, we use GBCV to assess LLM performance on them. Our findings indicate that GPT-4o performs better on complex program structures, while all models effectively detect boundary values in simple conditions but face challenges with arithmetic computations. This study highlights the strengths and limitations of LLMs in test generation, provides a benchmark framework, and suggests directions for future improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。