arXiv:2607.27030cs.CRcs.LG2026-07

用真实AI发现的漏洞构建测试集,验证代码分析器可靠性。

HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models

论文配图:HoF-Bench: Rediscovering Real AI-Discovered CVEs Without Frontier Models
图 1 · 摘自论文原文
  • 从95个真实漏洞中构建无标签测试集,仅提供源码和文件范围。
  • 最小化LLM分析器在严格标准下复现68%漏洞,证明其检测能力。
  • 适合评估漏洞检测工具稳定性与候选漏洞生成量,尤其关注C语言基础设施。

基于大模型的分析器已开始在成熟开源项目中发现真实漏洞:AISLE分析器在78个项目的OpenSSL、curl、GnuTLS等中被证实发现超过280个CVE。我们提出HoF-Bench(源自AISLE公开的荣誉榜),一个由8个仓库中95个公开AI发现的CVE组成的基准测试集,每个漏洞固定在特定的有缺陷提交版本。分析器仅获知源码和目标文件范围,无法获取CVE标识、描述、修复方式或预期机制;由不参与检测的前沿模型裁判仅认可那些识别出相同代码路径、根本原因、攻击条件和影响的发现。一个极简的LLM分析器在该严格协议下成功复现了95个漏洞中的65个(68%)。研究中所有前沿模型均未成功检测任何漏洞。测试包含十种检测器骨干:五种开源权重模型(总计21B–284B参数,激活部分3–13B)及五种专有的小型或“闪电”级模型。所有模型均运行于固定框架,含四轮重复执行、可选生成上下文阶段及可重放的多轮筛选阶段(共7,600次模型-漏洞运行记录)。漏洞难度显著受编程语言影响;所有模型均未能发现的漏洞集中于C语言基础设施代码。HoF-Bench为比较漏洞扫描器的性能、重复运行的可靠性及候选漏洞数量提供了紧凑测试平台。数据集已发布于 https://huggingface.co/datasets/aisleinc/HoF-Bench。

原文摘要 · Abstract (English)

LLM-based analyzers have begun finding real vulnerabilities in mature open-source projects: AISLE's analyzer is credited with more than 280 CVEs across 78 projects, including OpenSSL, curl, and GnuTLS. We introduce HoF-Bench (named after AISLE's public Hall of Fame), a benchmark built from 95 of these public AI-discovered CVEs across eight repositories pinned at vulnerable commits. Analyzers receive source and target-file scope but not CVE identifiers, descriptions, fixes, or expected mechanisms; a detector-blinded frontier-model judge credits only findings that identify the same code path, root cause, attack condition, and impact. A deliberately minimal LLM-based analyzer rediscovers up to 65 of the 95 CVEs (68%) under this strict protocol. No frontier model performs detection anywhere in the study. The ten detector backbones are five open-weight models (21B--284B total parameters, 3--13B active) and five proprietary small or "flash"-tier models. All of them run in the fixed scaffold with four repeated passes, an optional generated-context stage, and a replayable multi-round triage stage (7,600 model--CVE pass records). Difficulty is strongly structured by language; the CVEs missed by every model concentrate in C infrastructure code. HoF-Bench provides a compact test bed for comparing vulnerability scanners, their reliability across repeated runs, and the candidate volume they create. The dataset is available at https://huggingface.co/datasets/aisleinc/HoF-Bench.

漏洞检测LLM分析代码安全基准测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。