构建多语言漏洞检测基准，覆盖88个真实项目与48类漏洞。
VICBench: A Multi-Language Benchmark for Code Vulnerability Detection
- 通过人工与智能流程双重标注，生成100个经验证的漏洞引入提交（VIC）。
- 平均修复代码38.6行，漏洞引入代码252.5行，远超以往数据集。
- 适合评估漏洞检测算法，尤其对提升工具自动化水平有参考价值。
评估安全漏洞检测工具需要包含漏洞引入提交（VICs）的基准数据集——即首次在代码库中引入漏洞的提交。这些提交对于确定受影响软件版本的完整范围至关重要。现有漏洞数据集存在编程语言覆盖有限、补丁复杂度低、项目范围窄等问题。通过人类专家与代理工作流的双重标注，我们构建了包含100个经验证的VICs、对应100个CVE、覆盖Python、Java和C++三种语言的88个项目的基准数据集：VICBench，涵盖48种CWE类型。该数据集包含平均38.6行的复杂真实修复代码，以及对应平均252.5行的漏洞引入代码，显著大于先前工作。我们的评估表明，当前最先进的算法V-SZZ和LLM4SZZ仅达到33.3%-40.1%的F1值，证实现有方法仍需大量人工干预。VICBench为漏洞检测方法的稳健评估提供了支持。
原文摘要 · Abstract (English)
Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。