复现漏洞检测框架Vul-RAG,发现性能上限约30%准确率。
Revisiting Vul-RAG: Reproducibility and Replicability of RAG-based Vulnerability Detection with Open-Weight Models

- 在本地部署下复现Vul-RAG,使用开源模型验证结果
- 所有模型性能均止步于约30%的成对准确率
- 模型规模提升不带来显著效果,提示方法瓶颈存在
大型语言模型(LLMs)在自动化软件漏洞检测中展现出强大潜力,尤其在检索增强生成(RAG)场景中。然而,依赖专有模型和API的方法在可复现性和可重复性方面仍缺乏研究,其结果是否具有普遍性尚存疑问。本文对基于RAG的漏洞检测框架Vul-RAG进行了可复现性研究,首先在本地全开源权重环境下复现了原报告的基准结果。随后,评估了多类近期开源大模型,包括代码专用、通用及推理型模型,参数量各异。结果显示,尽管存在微小偏差,原结论可复现;所有模型在成对准确率上均达约0.30的平台期。即使采用更先进模型,该上限依然存在,表明仅提升模型容量无法显著改善性能。最后讨论了检测效果、模型能力与规模之间的权衡。实现与评估数据已公开于 https://github.com/hs-esslingen-it-security/revisiting-Vul-RAG。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown strong potential for automated software vulnerability detection, particularly in retrieval-augmented generation (RAG) settings. However, for approaches relying on proprietary models and APIs, reproducibility and replicability remain largely unexplored, raising the question of whether reported results generalize or depend primarily on specific model choices. In this work, we present a reproducibility study of Vul-RAG, a RAG-based framework for source code vulnerability detection that enhances LLMs with high-level vulnerability knowledge. We first replicate the results in a fully local and open-weights setting using the reported open-weight baseline models. We then extend the evaluation to a diverse set of recent open-weight LLMs, including code-specialized, general-purpose, and reasoning models of varying parameter sizes. The results confirm that the findings of Vul-RAG are reproducible under local deployment, but with minor deviations. Across all evaluated models, we observe a performance plateau at approximately 0.30 pairwise accuracy (code pairs for which both the vulnerable and the patched function are correctly classified). Notably, this plateau persists even for more recent and advanced models, indicating that improvements in model capacity alone do not substantially enhance performance. Finally, we discuss practical implications and trade-offs between detection effectiveness, model capabilities, and model scale. Implementation and evaluation artifacts are publicly available at https://github.com/hs-esslingen-it-security/revisiting-Vul-RAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。