Meta用即时捕获测试在代码合并前发现严重漏洞，减少误报提升效率。
Just-in-Time Catching Test Generation at Meta
- 通过感知代码变更生成候选捕获测试，效率比传统方法高4倍。
- 检测出8个真实漏洞，其中4个若未发现将导致严重故障。
- 结合规则与大模型评估，人工审查量减少70%，适合大型系统维护者。
我们报告了Meta在大规模后端系统（数亿行代码）中实施的即时捕获测试机制。与传统通过性加固测试不同，捕获测试旨在失败以暴露潜在缺陷。核心挑战是降低因误报导致的开发延迟。分析22,126个生成的测试表明，代码变更感知方法使候选捕获生成效率比加固测试高4倍，比偶然失败测试高20倍。为减少误报，采用基于规则和LLM的评估器，使人工审查工作量减少70%。统计分析显示，人类接受的代码变更被评估为更多误报，而被拒绝的变更则有显著更多真阳性。共上报41个候选捕获，其中8个被确认为真阳性，4个若未被捕获将引发严重故障。结果表明，即时捕获测试可规模化应用，有效防止严重问题进入生产环境。
原文摘要 · Abstract (English)
We report on Just-in-Time catching test generation at Meta, designed to prevent bugs in large scale backend systems of hundreds of millions of line of code. Unlike traditional hardening tests, which pass at generation time, catching tests are meant to fail, surfacing bugs before code lands. The primary challenge is to reduce development drag from false positive test failures. Analyzing 22,126 generated tests, we show code-change-aware methods improve candidate catch generation 4x over hardening tests and 20x over coincidentally failing tests. To address false positives, we use rule-based and LLM-based assessors. These assessors reduce human review load by 70%. Inferential statistical analysis showed that human-accepted code changes are assessed to have significantly more false positives, while human-rejected changes have significantly more true positives. We reported 41 candidate catches to engineers; 8 were confirmed to be true positives, 4 of which would have led to serious failures had they remained uncaught. Overall, our results show that Just-in-Time catching is scalable, industrially applicable, and that it prevents serious failures from reaching production.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。