构建首个面向Solidity的可执行评估基准,提升合约生成准确率。
Towards Automated Smart Contract Generation: Evaluation, Benchmarking, and Retrieval-Augmented Repair
- 用差分模糊测试构建2.8万+函数的真实合约基准集
- 发现合约生成失败主因是缺少内部状态信息,全上下文推理成本过高
- 提出检索增强修复框架,仅需最小片段即可修正错误,显著降本增效
智能合约主要以Solidity编写并部署于以太坊等区块链,部署后不可更改,因此功能正确性至关重要。现有Solidity代码生成评估多依赖表面指标(如BLEU、CrystalBLEU)或人工检查,与功能正确性相关性弱。相比Python,Solidity缺乏大规模执行级基准,制约了大模型在合约开发中的系统评估。本文提出SolBench,一个强调功能正确性的完整基准与自动化测试流水线,采用差分模糊测试。SolBench包含从Etherscan(genesis-2024)收集的7604个真实合约中提取的28825个函数,覆盖十类应用领域。我们对14种不同LLM(涵盖开源/闭源模型,参数量1.3B-671B,通用/代码专用架构)进行了基准测试。主要失败模式为缺失关键合约内信息,如状态变量和类型定义。提供完整合约上下文虽能提升准确率,但推理成本极高。为此,我们提出检索增强修复(RAR)框架,将编译器和运行时错误信息用于检索修复目标函数所需的最小合约片段,避免全上下文推理。该方法显著降低输入长度,同时提升功能正确性。进一步分析表明,RAR在检索与修复策略上均具一致增益。SolBench与RAR实现可解释的执行级评估与经济高效的代码生成改进。数据集与代码已公开于https://github.com/ZaoyuChen/SolBench。
原文摘要 · Abstract (English)
Smart contracts, predominantly written in Solidity and deployed on blockchains such as Ethereum, are immutable after deployment, making functional correctness critical. However, existing evaluations of Solidity code generation rely largely on surface-level metrics (e.g., BLEU, CrystalBLEU) or manual inspection, which correlate poorly with functional correctness. In contrast to Python, Solidity lacks large-scale, execution-based benchmarks, limiting systematic evaluation of large language models for smart contract development. We introduce SolBench, a comprehensive benchmark and automated testing pipeline for Solidity that emphasizes functional correctness via differential fuzzing. SolBench consists of 28825 functions extracted from 7604 real-world smart contracts collected from Etherscan (genesis-2024), spanning ten application domains. We benchmark 14 diverse LLMs, covering open and closed models, 1.3B-671B parameters, and both general-purpose and code-specialized architectures. The dominant failure mode is missing critical intra-contract information, such as state variables and type definitions. Providing full-contract context improves accuracy but incurs prohibitive inference costs. To address this, we propose Retrieval-Augmented Repair (RAR), a cost-effective framework that integrates execution feedback into code repair. RAR uses compiler and runtime error messages to retrieve only the minimal contract snippets needed to correct a target function, avoiding full-context inference. This significantly reduces input length while improving functional correctness. We further analyze retrieval and repair strategies within RAR, demonstrating consistent gains in accuracy and efficiency. SolBench and RAR enable principled, execution-based evaluation and economical improvement of Solidity code generation. Dataset and code are publicly available at https://github.com/ZaoyuChen/SolBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。