arXiv:2608.25158cs.AIcs.CR2026-08

用真实开源项目测试大模型找漏洞能力,更贴近实战。

FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs

论文配图:FuzzingBrain-Bench V1: Evaluating Open-Ended Bug Discovery by LLMs
图 1 · 摘自论文原文
  • 让大模型在真实项目中生成触发崩溃的输入,不依赖预设漏洞。
  • Claude Opus 4.8 在77个挑战中触发60个不同崩溃,得分196/579。
  • 适合研究大模型安全检测、自动化测试或漏洞挖掘的开发者。

评估大语言模型(LLMs)发现软件漏洞的能力日益重要。现有基准通常要求模型生成能触发预定义漏洞的验证输入,但这可能忽略模型发现但未匹配预设目标的有效崩溃。为此,我们提出 FuzzingBrain-Bench,一个用于评估 AI 模型在开源软件中发现漏洞能力的基准。模型在自包含 Docker 镜像中接收一个开源项目和经过沙箱工具(sanitizer)注入的测试程序(harness),目标是通过生成输入触发尽可能多的不同崩溃。每个挑战的得分基于模型产生的不同崩溃签名数量,上限为预设值,并按难度系数加权。FuzzingBrain-Bench V1 包含来自43个开源项目的77个挑战,其中36个为C语言,32个为C++,9个为Java/JVM。我们对 Claude Haiku 4.5、Claude Sonnet 4.6 和 Claude Opus 4.8 进行了全基准测试。Claude Opus 4.8 表现最佳,在77个挑战中触发60个不同崩溃,总得分为196/579。三款模型均未在13个挑战中触发崩溃。该基准数据集与测试程序已公开于 https://github.com/fuzzingbrain/FuzzingBrain-Bench。

原文摘要 · Abstract (English)

Evaluating the ability of large language models (LLMs) to discover software bugs is increasingly important. Existing benchmarks typically evaluate this capability by asking the model to generate a proof-of-concept input that triggers a predefined target vulnerability. However, this setup may overlook valid crashes discovered by the model when they do not match the predefined target. As a result, the evaluation may not reflect the model's real capability. We present FuzzingBrain-Bench, a benchmark for assessing AI models' ability to discover bugs in open-source software. Models are given an open-source project and a sanitizer-instrumented harness in a self-contained Docker image. Their goal is to generate inputs that trigger as many distinct crashes as possible through the harness. A model's performance on each challenge is scored based on the number of distinct crash signatures it produces, capped at a predefined maximum and weighted by a difficulty coefficient. FuzzingBrain-Bench V1 consists of 77 challenges drawn from 43 open-source projects, with 36 C, 32 C++, and 9 Java/JVM challenges. We evaluate Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8 on the full benchmark. Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579. None of the three models triggers a crash in 13 challenges. The FuzzingBrain-Bench corpus and harnesses are publicly available at https://github.com/fuzzingbrain/FuzzingBrain-Bench.

漏洞发现大模型测试代码安全自动化测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。