评测大模型在二进制逆向工程中的解题能力，聚焦可执行程序的逻辑还原与验证。
CrackMeBench: Binary Reverse Engineering for Agents
- 构建专用基准测试，评估模型通过工具逆向分析无符号二进制文件的能力。
- GPT-5.5 在生成任务中达成92%通过率，显著优于其他模型。
- 适合研究自主二进制分析、智能攻防或安全推理的开发者使用。
针对代码代理的基准测试越来越多地衡量源码修复能力，而网络安全基准则侧重于广义夺旗表现。传统二进制逆向工程仍缺乏精确规范：仅给定可执行文件，代理能否恢复验证逻辑并生成被程序接受的输入、序列号、产物或密钥生成器？我们提出 CrackMeBench，一个用于评估语言模型代理在教育类 CrackMe 风格逆向工程任务中的基准。该基准聚焦确定性二进制验证问题，具备可执行断言、符号极少的二进制文件、明确的本地工具访问及外部评分提交，而非自由格式解释。v0 版本包含八个公开校准 CrackMe 和十二个基于种子 C、Rust、Go 模板生成的主得分任务，所有模型在无网络的 Linux Docker 沙箱中通过统一外壳接口运行，配备标准逆向工具。三模型评估在五分钟预算内每任务三次提交，生成集上 pass@3 分别为：GPT-5.5 11/12（92%）、Claude Opus 4.7 7/12（58%）、Kimi K2 5/12（42%）。更难的生成半集进一步拉开差距，pass@3 为 5/6、2/6、1/6；在八任务公开校准集上，pass@3 为 3/8、2/8、1/8。CrackMeBench 记录 pass@1 与 pass@3、评分提交结果、耗时、命令追踪、工具类别、提供方报告的 token 使用量、估算成本及定性失败标签，为从源码推理迈向自主二进制分析的研究提供可复现的测试环境，同时限定范围为教育性、专设程序。
原文摘要 · Abstract (English)
Benchmarks for coding agents increasingly measure source-level software repair, and cybersecurity benchmarks increasingly measure broad capture-the-flag performance. Classical binary reverse engineering remains less precisely specified: given only an executable, can an agent recover validation logic and produce an input, serial, artifact, or key generator accepted by the program? We introduce CrackMeBench, a benchmark for evaluating language-model agents on educational CrackMe-style reverse-engineering tasks. CrackMeBench focuses on deterministic binary validation problems with executable oracles, symbol-poor binaries, explicit local tool access, and externally scored submissions rather than free-form explanations. The v0 benchmark combines eight public calibration CrackMes with twelve generated main-score tasks built from seeded C, Rust, and Go templates, and agents run through an equal shell interface in a no-network Linux Docker sandbox with standard reverse-engineering tools. In a three-model evaluation with a five-minute budget and three scored submissions per task, pass@3 on the generated split is 11/12 tasks (92%) for GPT-5.5, 7/12 (58%) for Claude Opus 4.7, and 5/12 (42%) for Kimi K2. The harder generated half separates the models more sharply, with pass@3 of 5/6, 2/6, and 1/6, respectively; on the eight-task public calibration split, pass@3 is 3/8, 2/8, and 1/8. CrackMeBench records pass@1 and pass@3, scored submissions, wall-clock time, command traces, tool categories, provider-reported token usage, estimated cost, and qualitative failure labels, providing a reproducible testbed for measuring progress from source-code reasoning toward autonomous binary analysis while restricting scope to educational, purpose-built programs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。