LLM测试生成工具因设计缺陷，反而掩盖代码错误。
Design choices made by LLM-based test generators prevent them from finding bugs
- 用人类编写的含错代码测试，发现生成的测试用例常无法暴露漏洞。
- 部分工具会误判错误代码为正确，导致测试套件包含虚假通过的测试。
- 适合关注AI测试可靠性与软件质量保障的研究者和开发者。
越来越多研究与商业工具采用大语言模型（LLMs）进行自动化测试用例生成。本文深入考察近期基于LLM的测试生成工具（如Codium CoverAgent和CoverUp）是否真正能发现缺陷，或无意中验证有缺陷的代码。由于缺陷仅在失败测试中显现，我们探讨：当测试断言设计为通过时，这些工具能否实现软件测试的初衷？以真实人类编写的含错代码为输入进行评估，结果显示LLM生成的测试常无法检测到缺陷，更严重的是，其设计可能导致错误代码被验证通过，并拒绝真正能揭示缺陷的测试用例。这些发现质疑了当前基于LLM测试生成工具的设计有效性，对软件质量和测试套件可靠性带来重要影响。
原文摘要 · Abstract (English)
There is an increasing amount of research and commercial tools for automated test case generation using Large Language Models (LLMs). This paper critically examines whether recent LLM-based test generation tools, such as Codium CoverAgent and CoverUp, can effectively find bugs or unintentionally validate faulty code. Considering bugs are only exposed by failing test cases, we explore the question: can these tools truly achieve the intended objectives of software testing when their test oracles are designed to pass? Using real human-written buggy code as input, we evaluate these tools, showing how LLM-generated tests can fail to detect bugs and, more alarmingly, how their design can worsen the situation by validating bugs in the generated test suite and rejecting bug-revealing tests. These findings raise important questions about the validity of the design behind LLM-based test generation tools and their impact on software quality and test suite reliability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。