arXiv:2506.11559cs.SEcs.AI2025-06被引 4

用GPT-4自动生成能验证漏洞修复的单元测试,降低安全测试成本。

Leveraging GPT-4 for Vulnerability-Witnessing Unit Test Generation

  • 基于漏洞代码前后对比,让GPT-4生成测试用例
  • 66.5%生成语法正确测试,7.5%可自动验证语义正确性
  • 生成结果易人工修正,适合安全工程师辅助使用

在软件开发生命周期中,测试对质量保障至关重要。充分的测试不仅能提升代码覆盖率、防止回归,还能识别并修复潜在漏洞。然而,手动编写测试耗时耗力。本文从漏洞角度,探索当前最广泛使用的大型语言模型GPT-4在自动单元测试生成方面的能力。研究基于VUL4J数据集中真实漏洞及其修复代码,考察GPT-4能否根据修复前后的代码生成语法或语义正确的单元测试,以作为漏洞缓解的证据。重点分析代码上下文影响、GPT-4自我纠错能力及生成测试的主观可用性。结果显示,未进行领域预训练时,GPT-4能生成66.5%语法正确的测试用例;仅7.5%的修复可通过自动化方式验证语义正确性。但主观评估表明,其生成的测试模板可经少量人工修改转化为完整有效的漏洞见证测试。因此,尽管数据有限,初步结果表明GPT-4可用于漏洞见证测试生成,虽无法完全自治,但在部分自动化流程中具有显著作用。

原文摘要 · Abstract (English)

In the life-cycle of software development, testing plays a crucial role in quality assurance. Proper testing not only increases code coverage and prevents regressions but it can also ensure that any potential vulnerabilities in the software are identified and effectively fixed. However, creating such tests is a complex, resource-consuming manual process. To help developers and security experts, this paper explores the automatic unit test generation capability of one of the most widely used large language models, GPT-4, from the perspective of vulnerabilities. We examine a subset of the VUL4J dataset containing real vulnerabilities and their corresponding fixes to determine whether GPT-4 can generate syntactically and/or semantically correct unit tests based on the code before and after the fixes as evidence of vulnerability mitigation. We focus on the impact of code contexts, the effectiveness of GPT-4's self-correction ability, and the subjective usability of the generated test cases. Our results indicate that GPT-4 can generate syntactically correct test cases 66.5\% of the time without domain-specific pre-training. Although the semantic correctness of the fixes could be automatically validated in only 7. 5\% of the cases, our subjective evaluation shows that GPT-4 generally produces test templates that can be further developed into fully functional vulnerability-witnessing tests with relatively minimal manual effort. Therefore, despite the limited data, our initial findings suggest that GPT-4 can be effectively used in the generation of vulnerability-witnessing tests. It may not operate entirely autonomously, but it certainly plays a significant role in a partially automated process.

漏洞检测自动化测试GPT-4代码生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。