用大模型预测测试用例会失败,提升找缺陷的准确性。
Fail-Aware and Explainable Test Oracle Prediction

- 直接预测测试前缀是否失败,而非生成断言。
- 在未见过的项目中,失败案例检测率显著高于基线方法。
- 可解释的语句级行为证据,适合集成到自动化测试流程中。
尽管测试断言在缺陷检测中至关重要,但有效构建仍具挑战性。现有基于学习的方法虽能自动生成断言,但常因无法揭示缺陷而失效。本文提出FOCAL,一种基于代码大模型的判别式断言预测器,不生成断言,而是直接预测给定测试前缀是否会失败。该模型从标注的测试前缀与被测方法对中学习,训练时强化失败样本的损失,并以语句级行为证据支持预测。相比基线方法SEER,FOCAL在未见项目上对失败案例的检测性能显著提升,且提供更丰富的解释。初步评估显示,在故障检测基准和自动生成测试数据集上,其在训练分布内高度准确,并在先前判别式断言失效的未见项目中大幅提升失败检测能力。被高亮的语句均通过行为解释验证。结果表明,该方法可补充模糊测试、搜索测试及大模型生成测试等技术,将大量生成的测试前缀转化为更可能暴露语义错误的可执行测试。
原文摘要 · Abstract (English)
Despite their central role in fault detection, test oracles remain challenging to construct effectively. Recent learning based methods address this challenge by automatically generating test assertions, yet even if syntactically correct, they are often ineffective in revealing bugs. Rather than generating assertions, this study explores a different approach by training a model to directly predict whether a given test prefix passes or fails. We present FOCAL, an emerging code LLM-based discriminative oracle predictor. It learns from labeled pairs of test prefixes and methods under test, employs losses that emphasize failing cases during training, and grounds its predictions in statement level behavioral evidence. Compared with the baseline method SEER, we substantially improve performance on failing cases for unseen projects and provide richer explanations. A preliminary evaluation on fault-detection benchmarks and automated test-generation artifacts shows that our approach is highly accurate within its training distribution and substantially improves failure detection on previously unseen projects where prior discriminative oracles collapse. Moreover, the highlighted statements are supported by behavioral explanation checks. These early results suggest that fail-aware discriminative oracle prediction can complement existing approaches such as fuzzing, search-based testing, and LLM-based test generation. These techniques produce test prefixes at scale but often lack fault oriented oracles. In future work, FOCAL could take generated test prefixes and attach fault-aware predicted oracles to them, turning high-volume input generation into executable tests that are more likely to expose semantic failures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。