实测四大代码漏洞检测模型,发现其在真实场景中表现大幅下滑。
From Lab to Reality: A Practical Evaluation of Deep Learning Models and LLMs for Vulnerability Detection
- 在四个数据集上独立训练并对比模型,分析代码表示的漏洞特征
- 新构建时序分离的外部数据集,模型在真实漏洞上准确率骤降
- 揭示学术基准与实际应用间巨大差距,强调真实评估的重要性
基于深度学习的漏洞检测方法在基准数据集上表现优异,但其在真实环境中的有效性尚未充分验证。本文系统评估了四种代表性模型(Devign、ReVeal、LineVul、VulBERTa)在四个典型数据集(Juliet、Devign、BigVul、ICVul)上的表现,并通过t-SNE和中心距分析其图结构与CodeBERT表示中的漏洞模式。为进一步评估实际适用性,研究在自建的时序分离外分布数据集VentiVul上测试了训练好的ReVeal、LineVul及四款开源大语言模型,该数据集包含200个来自Linux与Chromium的近期漏洞。实验显示,现有表示方法难以区分有无漏洞代码,模型在不同数据分布下泛化能力差;在VentiVul上性能急剧下降,多数模型无法可靠检测漏洞或识别修复函数。结果揭示了学术基准与真实部署间的持续鸿沟,凸显了构建面向部署的评估框架、改进代码表示、提升数据质量及考虑漏洞修复变化的必要性。
原文摘要 · Abstract (English)
Vulnerability detection methods based on deep learning (DL) have shown strong performance on benchmark datasets, yet their real-world effectiveness remains underexplored. Recent work suggests that graph neural network-based and transformer-based models, including large language models (LLMs), yield promising results when evaluated on curated benchmark datasets. These datasets are typically characterized by similar data distributions and may contain synthetic samples, heuristic labels, or labeling noise. In this study, we systematically evaluate four representative DL models---Devign, ReVeal, LineVul, and VulBERTa---across four representative datasets: Juliet, Devign, BigVul, and ICVul. Each model is trained independently on each dataset, and the graph-based and CodeBERT representations adopted by these models are analyzed using t-SNE and centroid distance to examine vulnerability-related patterns. To assess realistic applicability, we further evaluate trained ReVeal and LineVul models, along with four open-weight LLMs, on VentiVul, our newly constructed temporally separated out-of-distribution (OOD) dataset comprising 200 recent vulnerabilities from Linux and Chromium. Our experiments reveal that current representation methods struggle to distinguish vulnerable from non-vulnerable code and that trained models generalize poorly across datasets with differing distributions and characteristics. When evaluated on VentiVul, performance drops sharply, with most models failing to detect vulnerabilities reliably or distinguish vulnerable functions from their patched counterparts. These results expose a persistent gap between academic benchmarks and real-world deployment, emphasizing the value of our deployment-oriented evaluation framework and the need for more robust code representations, higher-quality datasets, and evaluation methods that account for vulnerability-fixing changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。