arXiv:2605.23243cs.CRcs.AI2026-05

顶尖大模型在网络安全检测中表现不佳,需专用垂直模型。

Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks

论文配图:Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks
图 1 · 摘自论文原文
  • 用白盒/黑盒双模式测试,发现大模型漏洞检测误报率高达10%-50%。
  • 黑盒测试仅覆盖4%-8%真实漏洞,即使结合工具也难超19%。
  • 专用安全模型精度达90.4%,证明方法比规模更重要。

我们通过双模式基准评估前沿大模型在网络安全中的适用性:白盒函数级漏洞检测(VulnLLM-R,涵盖C/Java/Python)和黑盒网页应用安全测试(五款生产级应用,含118个真实漏洞,覆盖20余种CWE类别,数据集将开源)。测试六款前沿模型(GPT-5.4、Codex~5.3、Claude Opus~4.7、Sonnet~4.6、Gemini~3.1~Pro、Gemini~3~Flash)及两款领域专用模型,覆盖四种测试范式。结果令人警醒:(1)所有前沿模型在白盒检测中误报率达10%-50%,系统性高估漏洞;(2)黑盒测试仅实现4%-8%的真实漏洞覆盖率,即使使用外部工具(Playwright MCP、Burp Suite MCP)提升至10%-19%;(3)领域专用智能体采用结构化渗透测试方法后,单类漏洞检测率超50%,表明方法优于模型规模;(4)领域专用防御模型在单张GPU上达到最高精度(0.904)与最低误报率(9.7%)。我们识别出训练数据缺乏端到端请求/响应序列、失败密集数据及多步攻击链是根本瓶颈,并提出自对弈安全测试作为数据生成策略。研究支持构建专为网络安全设计的垂直基础模型。

原文摘要 · Abstract (English)

We evaluate whether frontier LLMs are ready for cybersecurity through a dual-mode benchmark: white-box function-level vulnerability detection (VulnLLM-R, across C/Java/Python) and black-box web application security testing (five production-style applications with 118 ground-truth vulnerabilities across 20+ CWE families, which we will open-source). We test six frontier models (GPT-5.4, Codex~5.3, Claude Opus~4.7, Sonnet~4.6, Gemini~3.1~Pro and Gemini~3~Flash) and two domain-specialized models across four testing paradigms. Our findings are sobering: (1)~every frontier model produces 10-50% false positive rates in white-box detection, systematically over-predicting vulnerabilities; (2)~in black-box testing, frontier models achieve only 4-8% ground-truth coverage, improving to just 10-19% even with external security tools (Playwright MCP, Burp Suite MCP); (3)~structured penetration-testing methodology encoded in domain-specialized agents raises per-family detection above 50%, demonstrating that methodology, not scale, is the primary lever; and (4)~a domain-specialized defense model achieves the highest precision (0.904) and lowest false positive rate (9.7%) among all models, on a single GPU. We identify the absence of structured security testing traces end-to-end request/response sequences, failure-heavy data, and multi-step attack chains as the fundamental training data bottleneck, and propose self-play security testing as a data generation strategy. Our results make the case for vertical foundation models purpose-built for cybersecurity.

大模型安全漏洞检测垂直模型黑盒测试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。