为大模型生成的智能合约反编译建立评测基准,解决语义一致性难题。
SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
- 构建600个真实合约的反编译数据集,含字节码、源码与可重放语义检查点。
- 零样本测试显示顶尖模型仅42/600合约完全还原语义,编译修复可提升性能。
- 适合区块链安全、形式化验证与大模型代码生成研究者参考。
智能合约反编译旨在从字节码恢复高级源代码,但现有评估因数据集狭窄、指标不一和语义一致性不足而困难重重。随着大语言模型(LLMs)开始生成可编译且看似合理的Solidity代码,即使其语义已偏离原合约,这一问题愈发关键。本文提出SCDBench,一个面向基于大模型的智能合约反编译的数据集与评测方法。数据集包含600个真实世界Solidity合约,配有对应的字节码输入、真实源码及可重放的语义检查点。评估分四个累积阶段:格式完整性、可编译性、应用二进制接口(ABI)恢复以及通过差分重放实现的语义一致性。我们在零样本设置下评估Claude Opus 4.7、GPT-5.3-Codex和GLM-5,包括带与不带扩展推理的GLM-5变体,以及零样本编译修复设置。结果显示,前沿模型常能生成结构完整且可编译的Solidity,但语义一致性仍远未解决:最佳模型仅成功还原42/600合约的完整语义。进一步实验表明,引入同模型编译修复可显著提升性能,代价适中。SCDBench为可靠、可复现的评估提供统一基准,助力区块链安全与透明性发展。
原文摘要 · Abstract (English)
Smart contract decompilation aims to recover high-level source code from bytecode, but evaluating decompilers remains difficult because existing studies use narrow datasets, inconsistent metrics, and limited semantic consistency checks. This gap is increasingly important as large language models (LLMs) begin to generate source-like Solidity that may compile and appear plausible, even when its semantics diverge from the original contract. We introduce SCDBench, a dataset and benchmark methodology for LLM-based smart contract decompilation. The dataset contains 600 real-world Solidity contracts with paired bytecode inputs, ground-truth source code, and replayable semantic checkpoints. SCDBench evaluates decompiler outputs through four cumulative stages: format completeness, compilability, Application Binary Interface (ABI) recovery, and semantic consistency via differential replay. We evaluate Claude Opus 4.7, GPT-5.3-Codex, and GLM-5 in a zero-shot decompilation setting, including GLM-5 variants with and without extended reasoning and a zero-shot compilation-repair setting. The results show that frontier LLMs can often produce structured and compilable Solidity, but achieving semantic consistency remains far from solved: the best-performing frontier model perfectly decompiles only 42/600 contracts. We further show that introducing same-model compilation repair substantially improves performance at modest additional cost. SCDBench establishes a common ground for rigorous, reproducible evaluation and aims to accelerate the development of reliable smart contract decompilers for blockchain security and transparency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。