用真实链上漏洞测试AI agents在合约安全中的表现
CyberChainBench: Can AI Agents Secure Smart Contracts Against Real-World On-Chain Vulnerabilities?
- 构建包含541起真实事件的链上安全评测基准
- 最佳模型检测准确率37.5%，生成攻击获利达5740万美元
- 首次实现对修补方案的链上回放验证，适合安全研究者
我们提出CyberChainBench，一个用于评估基于大模型的智能体在智能合约安全方面表现的基准，涵盖漏洞检测、攻击生成和补丁合成三类任务。该基准基于DeFiHackLabs记录的541起真实世界漏洞事件，覆盖9条EVM链，通过Harbor搭建的隔离环境，让智能体与历史区块链状态交互，使用代码读取、交易追踪和主网分叉验证等工具进行端到端评估。每个案例锚定具体区块，包含漏洞类型、定位及攻击者收益的结构化真值。攻击按历史分叉上的经济影响评分，补丁通过在代理可升级子集上重放历史攻击与合法交易作为失败即通过的测试基准进行验证。定义五类漏洞分类法，评估多个模型-智能体组合。结果揭示明显难度梯度：最优配置在检测上得37.5%，攻击生成43.7%，补丁合成仅23.4%；顶级智能体（Codex + GPT-5.5）在200个案例中总获利5740万美元，单例成本2.39美元。
原文摘要 · Abstract (English)
We present CyberChainBench, a benchmark for evaluating LLM-based agents on smart contract security across three complementary tasks: vulnerability detection, exploit generation, and patch synthesis. Built from 541 real-world exploit incidents from DeFiHackLabs spanning 9 EVM chains, the benchmark provides end-to-end on-chain evaluation where agents interact with historical blockchain state through isolated evaluation environments orchestrated by Harbor, using tools to read code, trace transactions, and validate exploits on mainnet forks. Each case is anchored to a specific block and includes structured ground truth covering vulnerability type, localization, and attacker profit. Exploits are graded by economic impact on historical forks; patches are validated by replaying historical attacks and legitimate transactions as fail-to-pass test oracles on a proxy-upgradeable subset. We define a five-type vulnerability taxonomy and evaluate multiple agent--model configurations. Results reveal a clear difficulty gradient: the best configuration scores 37.5% on detection, 43.7% on exploitation, but only 23.4% on patching, with the top agent (Codex with GPT-5.5) realizing \$57.4M in total exploit profit across the 200-case exploit set at a cost of $2.39 per case.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。