LLM生成的测试在代码变更后失效严重，暴露其依赖表面模式而非真实语义理解。
Evaluating LLM-Based Test Generation Under Software Evolution
- 通过变异驱动框架，对比8个模型在2.2万多个代码变体下的测试表现
- 代码语义改变后测试通过率降至66%，分支覆盖率跌至60%
- 多数失败测试仍能通过原版代码，说明未真正适应新逻辑
大型语言模型（LLMs）被广泛用于自动化单元测试生成。然而，这些测试是否反映对程序行为的真实推理仍不明确，若仅依赖训练中习得的表面模式，则可能产生覆盖不足、漏检回归和错误无法发现等问题。我们开展了一项大规模实证研究，分析代码演化下LLM生成测试的表现。采用自动化变异驱动框架，评估8个LLM在22,374个程序变体上对语义改变（SAC）和语义保持改变（SPC）的响应。初始测试套件平均达到79%行覆盖和76%分支覆盖。但在语义改变下，新生成测试通过率降至66%，分支覆盖率下降至60%；超过99%的失败测试在原程序上执行修改区域仍能通过，表明其残留对原行为的依附性，而非适应新语义。即使功能不变的语义保持变化也导致测试通过率降至79%，分支覆盖率降至69%——因语法变动大，生成测试不稳定。模型倾向于生成更多新测试并丢弃原有测试，显示对词法变化敏感，而非真实语义影响。总体表明，当前LLM测试生成严重依赖表面线索，难以维持代码演化中的回归感知能力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used for automated unit test generation. However, it remains unclear whether these tests reflect genuine reasoning about program behavior or simply reproduce superficial patterns learned during training. If the latter dominates, LLM-generated tests may exhibit weaknesses such as reduced coverage, missed regressions, and undetected faults. Understanding how LLMs generate tests and how those tests respond to code evolution is therefore essential. We present a large-scale empirical study of LLM-based test generation under program changes. Using an automated mutation-driven framework, we analyze how generated tests react to semantic-altering changes (SAC) and semantic-preserving changes (SPC) across eight LLMs and 22,374 program variants. LLMs achieve strong baseline results, reaching 79% line coverage and 76% branch coverage with fully passing test suites on the original programs. However, performance degrades as programs evolve. Under SACs, the pass rate of newly generated tests drops to 66%, and branch coverage declines to 60%. More than 99% of failing SAC tests pass on the original program while executing the modified region, indicating residual alignment with the original behavior rather than adaptation to updated semantics. Performance also declines under SPCs despite unchanged functionality: pass rates fall to 79% and branch coverage to 69%. Although SPC edits preserve semantics, they often introduce larger syntactic changes, leading to instability in generated test suites. Models generate more new tests while discarding many baseline tests, suggesting sensitivity to lexical changes rather than true semantic impact. Overall, our results indicate that current LLM-based test generation relies heavily on surface-level cues and struggles to maintain regression awareness as programs evolve.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。