用语义保持变换提升代码缺陷检测工具效果,但实际效果有限。
Semantic-Preserving Transformations as Mutation Operators: A Study on Their Effectiveness in Defect Detection
- 将语义不变的代码变换作为突变算子,用于增强缺陷检测模型。
- 在Devign数据集上测试,三种集成策略均未提升模型准确率。
- 多数共享变换实际改变了语义,复用存在风险,适合工具开发者参考。
近期缺陷检测研究利用语言模型提升性能。现有方法通过增强训练数据提高模型对语义相同代码的鲁棒性(即预测结果应一致)。然而,尚未将语义相同的代码用于改进检测工具在实际应用阶段的表现——这与变异测试概念密切相关。本研究旨在验证:能否使用语义保持变换(类比突变算子)来提升缺陷检测工具在测试阶段的表现?我们首先收集了28篇实现语义保持变换的论文并共享其代码,共获得94种不同变换。从中选取39种进行实现,但人工检查发现其中23种实际改变了代码语义。最终使用剩余16种正确变换,结合三种集成策略,在Devign数据集上对VulBERTa和PLBART两个微调大语言模型进行测试,结果未能提升缺陷检测模型的准确率。研究显示,复用公开的语义保持变换存在困难,甚至可能引入错误语义变化。
原文摘要 · Abstract (English)
Recent advances in defect detection use language models. Existing works enhanced the training data to improve the models' robustness when applied to semantically identical code (i.e., predictions should be the same). However, the use of semantically identical code has not been considered for improving the tools during their application - a concept closely related to metamorphic testing. The goal of our study is to determine whether we can use semantic-preserving transformations, analogue to mutation operators, to improve the performance of defect detection tools in the testing stage. We first collect existing publications which implemented semantic-preserving transformations and share their implementation, such that we can reuse them. We empirically study the effectiveness of three different ensemble strategies for enhancing defect detection tools. We apply the collected transformations on the Devign dataset, considering vulnerabilities as a type of defect, and two fine-tuned large language models for defect detection (VulBERTa, PLBART). We found 28 publications with 94 different transformations. We choose to implement 39 transformations from four of the publications, but a manual check revealed that 23 out 39 transformations change code semantics. Using the 16 remaining, correct transformations and three ensemble strategies, we were not able to increase the accuracy of the defect detection models. Our results show that reusing shared semantic-preserving transformation is difficult, sometimes even causing wrongful changes to the semantics. Keywords: defect detection, language model, semantic-preserving transformation, ensemble
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。