新基准ULB挑战大模型真实函数测试生成能力
Benchmarking LLMs for Unit Test Generation from Real-World Functions
- 构建高复杂度真实代码库，避免测试数据泄露
- 大模型在新基准上准确率仅41.32%，显著低于旧基准
- 适合评估大模型真实场景推理能力，非单纯记忆
大语言模型（LLMs）在自动化单元测试生成方面展现出巨大潜力，可大幅减少开发者手动工作量。然而，现有测试生成基准存在数据污染和代码结构过于简单两大缺陷，导致研究结论不可靠且难以泛化。为此，我们提出ULB（UnLeakedTestbench），一个专为真实世界Python函数设计的函数级测试生成基准。通过多阶段筛选流程，确保高圈复杂度并有效防止测试用例泄露。该基准包含3,909个精心挑选的任务，更具现实挑战性。同时提供配套的PLT（PreLeakedTestbench）用于控制分析模型记忆与推理行为。实验表明，所有模型在ULB上的平均准确率为41.32%，语句覆盖率45.10%，分支覆盖率30.22%，变异分数40.21%，显著低于TestEval（91.79%、92.18%、82.04%、49.69%）和PLT（47.07%、55.13%、40.07%、50.80%）。
原文摘要 · Abstract (English)
Recently, large language models (LLMs) have shown great promise in automating unit test generation, significantly reducing the manual effort required by developers. To effectively evaluate the capabilities of LLMs in this domain, it is crucial to have a well-designed benchmark that accurately reflects real-world scenarios and mitigates common pitfalls. Existing LLM test generation benchmarks are limited by two critical drawbacks: data contamination and structurally simple function code. As a result, we often cannot rely on the validity of scientific conclusions drawn from empirical studies using these limited benchmarks. The empirical evidence presented may be biased due to contamination and may fail to generalize beyond toy programs due to structural simplicity. To address these problems, we introduce ULT (UnLeakedTestbench), a new benchmark specifically designed for function-level unit test generation from real-world Python functions. ULT is constructed through a multi-stage curation process that ensures high cyclomatic complexity and mitigates test case contamination. With 3,909 carefully selected function-level tasks, ULT provides a more realistic and challenging evaluation of LLMs' test generation capabilities. We also provide PLT (PreLeakedTestbench), a pair benchmark of ULT with leaked tests designed to enable a controlled analysis of memorization versus reasoning in test generation. Our evaluation results demonstrate that ULT is significantly more challenging. For example, test cases generated by LLMs only achieve 41.32\%, 45.10\%, 30.22\%, and 40.21\% for accuracy, statement coverage, branch coverage, and mutation score on average for all LLMs, respectively. These results are substantially lower than the corresponding metrics on TestEval (91.79\%, 92.18\%, 82.04\%, and 49.69\%) and PLT (47.07\%, 55.13\%, 40.07\%, and 50.80\%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。