LLM生成的测试用例中，覆盖率与变异得分能否预测真实漏洞检测效果？
Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)
- 在无缺陷代码场景下，覆盖率和变异分可有效比较不同模型性能
- 在已有缺陷代码中，这些指标无法可靠反映漏洞检测能力
- 测试规模不是主要干扰因素，评估需结合具体场景
大语言模型（LLMs）推动了测试自动生成的发展，现有研究多采用代码覆盖率和变异得分等代理指标评估生成测试集。然而，Inozemtseva 等人与 Papadakis 等人的研究表明，对于人工编写的测试，在控制测试集规模后，覆盖率、变异得分与真实漏洞检测能力的相关性可能消失，引发对代理指标有效性质疑。当前尚不清楚该结论是否适用于 LLM 生成的测试，因为其工作流与传统方法存在显著差异。本文通过大规模复制研究，使用多种 LLM 生成的广泛测试集，重新检验覆盖率、变异得分与真实漏洞检测之间的关系。结果表明：覆盖率与变异得分的有效性高度依赖上下文——在回归式场景（代码可假设无缺陷）中，它们能提供有意义的模型间比较信号；但在代码本身可能存在缺陷、目标是暴露其中漏洞的场景下，它们不再可靠。此外，我们发现测试集规模对三者相关性的影响较小。基于此，本文讨论了以往研究结果的解释方式，并为评估 LLM 测试生成提供了可操作建议。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have driven growing interest in using LLMs to automate test generation. Prior work commonly evaluates generated test suites using proxy metrics such as code coverage and mutation score. However, studies by Inozemtseva et al. and Papadakis et al. show that, for human-written tests, correlations among coverage, mutation, and real-bug detection can largely vanish once test suite size is controlled, raising concerns about the validity of evaluations based on proxy metrics. It also remains unclear whether these conclusions carry over to LLM-generated tests, given that prevailing LLM-based test-generation workflows differ substantially from traditional approaches. In this paper, we conduct a large-scale replication study of these two prior works using a wide range of test suites generated by a diverse set of LLMs, and re-examine the relationships among coverage, mutation, and real-bug detection effectiveness. Our findings diverge substantially from prior results. We show that the usefulness of coverage and mutation is highly context-dependent: in regression-style settings where the code provided to the LLM can be reasonably assumed bug-free, these metrics can provide meaningful signals when comparing across models; in another common scenario where the code-under-test may already be buggy and the goal is to expose the bug within the code-under-test, they no longer serve as reliable indicators. We also find little evidence that test suite size is a dominant confounder for correlations among coverage, mutation, and real-bug detection for LLM-generated tests. Based on these findings, we discuss how to interpret results from prior studies and provide actionable guidance for evaluating LLM-based test generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。