数据质量与模型选择决定AI漏洞检测能否跨项目泛化。
Data and Context Matter: Towards Generalizing AI-based Software Vulnerability Detection
- 构建高质量数据集VulGate,去除错误标注和重复样本
- 编码器模型在准确率和跨项目泛化上提升6.8%召回率
- 适合关注软件安全与AI泛化能力的研究者
基于AI的漏洞检测方法在特定代码库中表现优异,但难以泛化到未见代码。本文系统研究模型架构、参数配置及训练数据质量对泛化能力的影响。提出VulGate——一个高质量基准数据集,通过清理误标与重复样本、更新新漏洞、增加元数据、引入难例样本并设立专用测试集,克服了以往数据集的缺陷。实验表明,数据多样性和质量提升显著增强检测效果。同时对比多个仅编码器与仅解码器模型,发现编码器类模型在准确率与泛化性能上更优。所提模型在BigVul基准上实现6.8%的召回率提升,并在未见项目中表现更佳,验证了更强的跨项目泛化能力。结果强调数据质量与模型选型对构建稳健漏洞检测系统的关键作用,为未来具备高跨项目有效性的系统提供方向。
原文摘要 · Abstract (English)
AI-based solutions demonstrate remarkable results in identifying vulnerabilities in software, but research has consistently found that this performance does not generalize to unseen codebases. In this paper, we specifically investigate the impact of model architecture, parameter configuration, and quality of training data on the ability of these systems to generalize. For this purpose, we introduce VulGate, a high quality state of the art dataset that mitigates the shortcomings of prior datasets, by removing mislabeled and duplicate samples, updating new vulnerabilities, incorporating additional metadata, integrating hard samples, and including dedicated test sets. We undertake a series of experiments to demonstrate that improved dataset diversity and quality substantially enhances vulnerability detection. We also introduce and benchmark multiple encoder-only and decoder-only models. We find that encoder-based models outperform other models in terms of accuracy and generalization. Our model achieves \textbf{6.8\%} improvement in recall on the benchmark BigVul dataset and outperforms others on unseen projects, demonstrating enhanced generalizability. Our results highlight the role of data quality and model selection in the development of robust vulnerability detection systems. Our findings suggest a direction for future systems with high cross-project effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。