首个评估AI攻防能力对真实系统影响的框架，量化了漏洞发现与修复的经济价值。
BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems
- 构建25个真实系统，设计检测、利用、修复三类任务评估AI攻防能力。
- 顶级模型在修复任务中成功率超90%，对应最高3万美金收益，防御能力普遍强于攻击。
- 首次将漏洞生命周期与实际奖金挂钩，适合安全研究与模型评估者参考。
AI代理可能显著改变网络安全格局。本文提出首个框架，用于捕捉演化中真实系统的攻防能力。基于BountyBench，我们搭建了25个具有复杂真实代码库的系统，定义三类任务：检测（发现新漏洞）、利用（利用已知漏洞）和修补（修复已知漏洞）。针对检测任务，设计了一种跨漏洞类型的通用成功指标，实现局部评估。每个系统均手动配置环境，包括安装依赖、部署服务器及填充数据库。共设置40个漏洞赏金，金额从10美元至30,485美元不等，覆盖OWASP Top 10中的9项风险。通过信息引导策略调节任务难度，从零日漏洞识别到已知漏洞利用进行插值。评估10个代理：Claude Code、OpenAI Codex CLI（o3-high 和 o4-mini）、自研代理（o3-high、GPT-4.1、Gemini 2.5 Pro Preview、Claude 3.7 Sonnet Thinking、Qwen3 235B A22B、Llama 4 Maverick、DeepSeek-R1）。在最多三次尝试下，表现最优的为Codex CLI: o3-high（检测成功率12.5%，对应3,720美元；修补90%，对应14,152美元），自研代理：Claude 3.7 Sonnet Thinking（利用67.5%），Codex CLI: o4-mini（修补90%，对应14,422美元）。Codex CLI: o3-high、o4-mini及Claude Code在防御上更优，修补成功率分别为90%、90%、87.5%，而利用成功率分别为47.5%、32.5%、57.5%；自研代理攻防相对均衡，利用成功率17.5%-67.5%，修补成功率25%-60%。
原文摘要 · Abstract (English)
AI agents have the potential to significantly alter the cybersecurity landscape. Here, we introduce the first framework to capture offensive and defensive cyber-capabilities in evolving real-world systems. Instantiating this framework with BountyBench, we set up 25 systems with complex, real-world codebases. To capture the vulnerability lifecycle, we define three task types: Detect (detecting a new vulnerability), Exploit (exploiting a given vulnerability), and Patch (patching a given vulnerability). For Detect, we construct a new success indicator, which is general across vulnerability types and provides localized evaluation. We manually set up the environment for each system, including installing packages, setting up server(s), and hydrating database(s). We add 40 bug bounties, which are vulnerabilities with monetary awards from \$10 to \$30,485, covering 9 of the OWASP Top 10 Risks. To modulate task difficulty, we devise a new strategy based on information to guide detection, interpolating from identifying a zero day to exploiting a given vulnerability. We evaluate 10 agents: Claude Code, OpenAI Codex CLI with o3-high and o4-mini, and custom agents with o3-high, GPT-4.1, Gemini 2.5 Pro Preview, Claude 3.7 Sonnet Thinking, Qwen3 235B A22B, Llama 4 Maverick, and DeepSeek-R1. Given up to three attempts, the top-performing agents are Codex CLI: o3-high (12.5% on Detect, mapping to \$3,720; 90% on Patch, mapping to \$14,152), Custom Agent: Claude 3.7 Sonnet Thinking (67.5% on Exploit), and Codex CLI: o4-mini (90% on Patch, mapping to \$14,422). Codex CLI: o3-high, Codex CLI: o4-mini, and Claude Code are more capable at defense, achieving higher Patch scores of 90%, 90%, and 87.5%, compared to Exploit scores of 47.5%, 32.5%, and 57.5% respectively; while the custom agents are relatively balanced between offense and defense, achieving Exploit scores of 17.5-67.5% and Patch scores of 25-60%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。