构建分层漏洞利用评估基准,揭示大模型真实安全攻防能力差距。
ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents

- 将漏洞利用分解为16个可测能力阶段,从触发崩溃到任意代码执行
- 41个V8漏洞测试中,公开模型仅半数能实现任意代码执行
- 适合研究大模型安全攻防、智能渗透测试的开发者与安全研究人员
漏洞利用并非二元事件,而是一个逐步获取能力的过程,从执行一行有缺陷的代码到完全控制目标系统。现有大模型安全评测将程序崩溃视为利用成功,这掩盖了从触发漏洞到构建可复用原语和控制权的关键难点。本文提出ExploitBench,一个分能力层级的评测基准,将利用过程拆解为16个可测量的标志:覆盖范围、崩溃、沙箱原语、任意读写、控制流劫持及任意代码执行。每个能力由确定性验证器检测:使用每轮随机化的挑战-响应机制验证原语,通过与真实二进制文件的差分执行衡量进展,以信号处理器证明代码执行。我们在41个V8漏洞上实例化该基准,设置三组实验:(<model,env>)主实验,评估模型-环境能力;(<model,env,adaptive coaching>)加入自适应指导,测试反馈是否改变结果;(<model,env,harness>)替换为模型原生命令行接口,检验厂商侧优化是否提升能力。结果显示,8个公开部署的前沿模型均能到达漏洞代码并引发崩溃,但仅有少数可实现任意代码执行;而私有模型在约一半案例中达成任意代码执行。总体表明,针对加固目标的漏洞构造是当前大模型尚未成熟的一项前沿能力。
原文摘要 · Abstract (English)
Exploitation is not a binary event. It is a ladder of acquiring progressive capabilities, from executing a single buggy line of code to taking full control of the target. However, existing LLM security benchmarks treat a crash as exploitation success. That single binary outcome collapses the hard parts of exploitation: the transition from triggering a bug to constructing reusable primitives and control. We present ExploitBench, a capability-graded benchmark that decomposes exploitation into 16 measurable flags, from coverage and crash through sandbox primitives, arbitrary read/write, control-flow hijack, and arbitrary code execution. Each capability is verified by a deterministic oracle that uses a per-run randomized challenge-response for primitives, differential execution against ground-truth binaries to measure progress, and a signal-handler proof for code execution. We instantiate ExploitBench on 41 V8 bugs because V8 is both widely deployed and exploitation-hardened. We report three arms: <model,env> as the primary measurement of model-environment capability, <model,env, adaptive coaching> as a secondary arm that adds adaptive coaching to test whether targeted feedback shifts outcomes, and <model,env,harness> as an ablation that swaps in the model's native CLI to check whether vendor-side optimizations increase exploitation capabilities. Our results show a sharp capability split between publicly deployed frontier models and the private frontier. Across the 8 publicly deployed models tested, reaching the vulnerable code and triggering a crash is routine, but arbitrary code execution is not. The private model shows arbitrary code execution on approximately half. Overall, results suggest that exploit construction against hardened targets is an emerging frontier capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。