用少量测试用例高效发现生成程序中的隐藏缺陷。
FaultLens: Learning Compact Behavioral Test Suites for Generated Operational Programs

- 基于历史生成程序学习测试探针排序,动态覆盖多种故障场景。
- 仅用1.2%-2.0%的测试量覆盖99.0%可检测到的故障。
- 适合需要可靠、可审计测试的生成式系统部署场景。
生成的操作程序通常通过少量人工编写用例或全量回归测试验证,前者易遗漏边界和交互错误,后者成本过高。我们提出FaultLens,一种学习紧凑行为测试集的方法,同时保留执行证据的可审计性。该方法仅需执行一次丰富探针域,将故障探测杀死关系存为稀疏结果缓存,并仅从早期程序生成中学习探针排序。故障驱动的贪心组件利用已知的杀死结构,而与变异无关的多样性组件覆盖探针族、案例、模板和时间区间。两者交替混合,在新程序包含训练阶段未见的故障机制时仍有效。我们在四个环境、十次执行种子、1,200个运行摘要、2,160次受控程序变换及4,120,200个程序-探针对上评估。在1,960个预设故障变换中,1,779个改变了有限审计域内的合约或输出;200个额外控制保持行为不变。在第1-3代上训练的32探针混合策略,覆盖了第4-5代中582个动态可杀故障中的576个(99.0%),仅使用全量域的1.2%-2.0%。当一个完整故障族被排除训练时,多样性使场景族宏观覆盖率从84.6%提升至94.9%。下游部署研究显示,保守准入规则将严重尾部退化从20组中的15组降至0/20。FaultLens提供优先级证据机制,而非正确性证明,其预算、证据来源、泛化拆分和遗漏均明确透明。
原文摘要 · Abstract (English)
Generated operational programs are often validated with either a few hand-written examples or exhaustive regression suites. The former can miss sparse boundary and interaction faults, while the latter can be unnecessarily expensive. We introduce FaultLens, a method for learning compact behavioral test suites while preserving an auditable connection to executed evidence. It executes a rich probe domain once, stores the fault-probe kill relation as a sparse outcome cache, and learns probe orderings only from earlier program generations. A fault-driven greedy component exploits known kill structure, while a mutation-independent diversity component covers probe families, cases, templates, and temporal bins. Their alternating hybrid remains useful when a new program contains a fault mechanism absent from ordering construction. We evaluate twenty generated operational policies across four environments, ten execution seeds, 1,200 measured run summaries, 2,160 controlled program transformations, and 4,120,200 executed program-probe pairs. Of 1,960 intended faulty transformations, 1,779 alter a contract or output somewhere in the finite audit domain; 200 additional controls preserve behavior. A 32-probe hybrid learned on generations 1-3 covers 576/582 (99.0%) dynamically killable faults in generations 4-5 using 1.2-2.0% of the exhaustive domain. With an entire fault family withheld from training, diversity raises scenario-family macro coverage from 84.6% to 94.9%. In a downstream deployment study, a conservative admission rule reduces severe tail regressions from 15/20 program-environment groups to 0/20. FaultLens provides a prioritized evidence mechanism, not a proof of correctness, and makes its budget, evidence source, generalization split, and misses explicit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。