首个真实且无污染的二进制逆向工程基准,测试大模型真逆向能力。
The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark

- 从零构建19个真实规模程序,用44种反分析技术生成262个二进制实例
- 最强模型仅61.4%任务正确,31.5%完整解决,说明逆向仍远未突破
- 专为防止模型作弊设计,适合评估大模型在真实软件逆向中的表现
AI代理在有源码时已展现强大安全能力,但多数关键软件如恶意代码、固件和专有应用仅以二进制形式存在。分析此类软件需逆向工程(RE):恢复程序语义后方可有效分析。然而,评估智能体逆向面临根本挑战:基准必须在训练数据中未出现以防止模型“认出”而非真正分析,同时需具备真实软件的规模与防护强度。现有基准无法兼顾二者。为此,我们提出SRE-Bench,首个真实、无污染的逆向工程基准。由逆向专家耗时超5000小时从零构建,包含19个私有真实规模程序,平均16.9K行代码。我们开发44种自研反分析原语,生成262个二进制实例和1572个可确定性评分的任务。对五款前沿大模型(GPT-5.6-sol、Claude-Opus-5、GPT-5.5、Grok-4.5、GLM-5.2)的评估显示,逆向仍基本未解:最强模型GPT-5.6-sol每实例准确率61.4%,仅31.5%实例被完全解决。分析还发现,智能体行为与人类工程师不同,对编译优化和静态链接不敏感。受控消融实验确认,污染控制与真实规模均至关重要。结果表明,源码安全能力尚未迁移到二进制分析,凸显逆向是智能体网络安全的关键前沿,SRE-Bench则是衡量进展的严格基准。
原文摘要 · Abstract (English)
AI agents are rapidly improving in cybersecurity capabilities when the source code is available for analysis, yet much of the software most consequential to cybersecurity, including malware, firmware, and proprietary applications, is available only as binaries. Analyzing such software requires reverse engineering(RE): recovering program semantics before the analysis can be meaningfully performed. However, evaluating agentic RE poses a fundamental challenge: benchmark instances must be unseen as source code in the LLMs' training data to prevent models from taking shortcuts by recognizing them rather than really analyzing them, while also matching the scale and anti-analysis protections of real software. Unfortunately, however, existing benchmarks do not jointly satisfy these requirements. To this end, we introduce SRE-Bench, the first realistic, contamination-free RE benchmark. Built entirely from scratch by RE experts with over 5,000 hours, SRE-Bench comprises 19 private, real-world-scale programs averaging 16.9K lines of code. We further developed 44 in-house anti-analysis primitives, yielding 262 binary instances and 1572 deterministically graded tasks. Our evaluation across five frontier LLMs (GPT-5.6-sol,Claude-Opus-5,GPT-5.5,Grok-4.5, and GLM-5.2) shows that RE remains largely unsolved: the strongest model, GPT-5.6-sol, scores 61.4% per instance, and fully solves only 31.5% of the instances. Our analysis further reveals that agents behave differently from human engineers, where agents are relatively insensitive to compiler optimization and static linking. Controlled ablations also confirm that both contamination control and realistic scale are essential. These results indicate that strong source-code security capabilities do not yet transfer to binary analysis, highlighting RE as an important frontier for agentic cybersecurity and SRE-Bench as a rigorous testbed to measure progress.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。