重新评估18种预训练模型在漏洞检测中的表现,发现其实际应用仍有局限。
Revisiting Pre-trained Language Models for Vulnerability Detection
- 系统评估18个预训练模型在真实代码数据上的表现
- 发现模型在复杂依赖漏洞上检测能力不足
- 适合关注代码安全与模型可靠性的研究人员
预训练语言模型在代码相关任务中表现出色,但在真实漏洞检测场景中的有效性仍存疑。现有研究普遍存在数据泄露、范围有限和分析肤浅等问题。本文通过严谨的评估流程,对18种模型在高质量数据集(含准确标注、多样漏洞类型、多项目数据)上进行深入评估。比较了微调与提示工程的效果,分析了不同训练-测试设置下的泛化能力,以及对代码归一化、抽象等扰动的鲁棒性。结果表明,专为代码语法语义设计预训练任务的模型优于通用模型或仅在大规模代码语料上预训练/微调的模型。但这些模型在复杂依赖漏洞检测、语义保持的代码变换识别,以及受限上下文窗口导致的标注错误方面仍面临挑战。本研究强调了在真实场景下全面评估模型性能的重要性,并指明未来改进方向。
原文摘要 · Abstract (English)
The rapid advancement of pre-trained language models (PLMs) has demonstrated promising results for various code-related tasks. However, their effectiveness in detecting real-world vulnerabilities remains a critical challenge. While existing empirical studies evaluate PLMs for vulnerability detection (VD), they suffer from data leakage, limited scope, and superficial analysis, hindering the accuracy and comprehensiveness of evaluations. This paper begins by revisiting the common issues in existing research on PLMs for VD through the evaluation pipeline. It then proceeds with an accurate and extensive evaluation of 18 PLMs on high-quality datasets that feature accurate labeling, diverse vulnerability types, and various projects. Specifically, we compare the performance of PLMs under both fine-tuning and prompt engineering, assess their effectiveness and generalizability across various training and testing settings, and analyze their robustness to a series of perturbations. Our findings reveal that PLMs incorporating pre-training tasks designed to capture the syntactic and semantic patterns of code outperform both general-purpose PLMs and those solely pre-trained or fine-tuned on large code corpora. However, these models face notable challenges in real-world scenarios, such as difficulties in detecting vulnerabilities with complex dependencies, handling perturbations introduced by code normalization and abstraction, and identifying semantic-preserving vulnerable code transformations. Also, the truncation caused by the limited context windows of PLMs can lead to a non-negligible number of labeling errors, which is overlooked by previous work. This study underscores the importance of thorough evaluations of model performance in practical scenarios and outlines future directions to help enhance the effectiveness of PLMs for realistic VD applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。