arXiv:2512.22306cs.CRcs.AI2025-12被引 2

首个针对多漏洞检测的综合基准,揭示大模型在复杂代码中的性能骤降。

Beyond Single Bugs: Benchmarking Large Language Models for Multi-Vulnerability Detection

  • 构建4万份带可控漏洞数的代码样本,覆盖四大语言
  • 漏洞密度越高,模型表现越差,最高下降40%
  • Python/JS模型严重漏检,召回率低至0.3

大型语言模型(LLMs)在自动化软件安全领域展现出巨大潜力,尤其在漏洞检测方面。然而,现有基准多聚焦于孤立的单漏洞样本或函数级分类,未能反映真实世界软件中多个相互关联漏洞共存的复杂性。近期研究表明,LLMs在多标签任务中存在‘计数偏差’和‘选择偏差’,但这一问题在代码安全领域尚未得到严格量化。本文提出一个涵盖C、C++、Python和JavaScript四种主要语言的多漏洞检测综合基准。通过从CodeParrot获取长上下文代码样本(7.5k-10k tokens),系统性注入1、3、5和9个漏洞,构建包含4万份文件的数据集。评估了GPT-4o-mini、Llama-3.3-70B及Qwen-2.5系列等五款先进LLM。结果表明,随着漏洞密度上升,模型性能显著下降:尽管Llama-3.3-70B在单漏洞C任务中F1得分接近0.97,但在高密度场景下性能最多下降40%。值得注意的是,Python与JavaScript表现出不同于C/C++的失效模式,模型在复杂Python文件中出现严重‘漏检’,召回率降至0.3以下。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have demonstrated significant potential in automated software security, particularly in vulnerability detection. However, existing benchmarks primarily focus on isolated, single-vulnerability samples or function-level classification, failing to reflect the complexity of real-world software where multiple interacting vulnerabilities often coexist within large files. Recent studies indicate that LLMs suffer from "count bias" and "selection bias" in multi-label tasks, yet this has not been rigorously quantified in the domain of code security. In this work, we introduce a comprehensive benchmark for Multi-Vulnerability Detection across four major languages: C, C++, Python, and JavaScript. We construct a dataset of 40,000 files by systematically injecting controlled counts of vulnerabilities (1, 3, 5, and 9) into long-context code samples (7.5k-10k tokens) sourced from CodeParrot. We evaluate five state-of-the-art LLMs, including GPT-4o-mini, Llama-3.3-70B, and the Qwen-2.5 series. Our results reveal a sharp degradation in performance as vulnerability density increases. While Llama-3.3-70B achieves near-perfect F1 scores (approximately 0.97) on single-vulnerability C tasks, performance drops by up to 40% in high-density settings. Notably, Python and JavaScript show distinct failure modes compared to C/C++, with models exhibiting severe "under-counting" (Recall dropping to less than 0.30) in complex Python files.

漏洞检测大模型多漏洞代码安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。