用大模型生成即时测试，提前捕获代码漏洞。
Harden and Catch for Just-in-Time Assured LLM-Based Software Testing: Open Research Challenges
- 提出硬化工件与即时捕获测试的定义与生成方法
- 实验证明可提前发现未来回归缺陷，提升测试效率
- 适合关注AI辅助测试、持续集成的开发者
尽管自动化软件测试研究已历时数十年，但若干基础概念仍定义不清且探索不足，却具备巨大实际应用潜力。本文在大语言模型（LLM）驱动的测试生成背景下，正式定义并研究了硬化工件测试与捕获测试的特性。硬化工件测试旨在防范未来回归问题，而捕获测试则用于发现代码变更引入的新故障或已有缺陷。硬化工件测试可在任意时间生成，一旦捕获到回归，即转化为捕获测试。本文提出‘即时捕获’（Just-in-Time, JiTTest）挑战，即在代码提交前即时生成测试以捕捉新缺陷。我们证明，解决此挑战的方法亦可用于发现遗留代码中的潜在缺陷。论文列举了硬化工件、捕获测试与JiTTest可能的演化路径，讨论开放研究问题、部署方案及在Meta开展的自动化LLM硬化工件生成初步成果。
原文摘要 · Abstract (English)
Despite decades of research and practice in automated software testing, several fundamental concepts remain ill-defined and under-explored, yet offer enormous potential real-world impact. We show that these concepts raise exciting new challenges in the context of Large Language Models for software test generation. More specifically, we formally define and investigate the properties of hardening and catching tests. A hardening test is one that seeks to protect against future regressions, while a catching test is one that catches such a regression or a fault in new functionality introduced by a code change. Hardening tests can be generated at any time and may become catching tests when a future regression is caught. We also define and motivate the Catching 'Just-in-Time' (JiTTest) Challenge, in which tests are generated 'just-in-time' to catch new faults before they land into production. We show that any solution to Catching JiTTest generation can also be repurposed to catch latent faults in legacy code. We enumerate possible outcomes for hardening and catching tests and JiTTests, and discuss open research problems, deployment options, and initial results from our work on automated LLM-based hardening at Meta. This paper was written to accompany the keynote by the authors at the ACM International Conference on the Foundations of Software Engineering (FSE) 2025. Author order is alphabetical. The corresponding author is Mark Harman.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。