构建首个评估LLM生成智能合约质量的基准，验证其可靠性并揭示典型错误模式。
SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications
- 基于自然语言生成合约，设计五维评分体系
- 9000份合约中35.3%存在逻辑遗漏，23.4%状态转移出错
- 适合安全研究者与区块链开发人员参考
我们提出SmartEval，一个系统性评估大语言模型（LLMs）从自然语言规范生成Solidity智能合约质量的基准。该基准包含9000个生成合约及其来自FSMSCG数据集的专家编写真实实现，采用涵盖功能完整性、变量一致性、状态机正确性、业务逻辑一致性和代码质量的五维评估体系，并提供可复现的生成与评估流程。通过三项独立实证研究验证其可靠性：五组条件消融实验（每组N=300）分离各组件贡献；三位哥伦比亚大学博士专家评估显示自动化评分与人工判断偏差小于0.34分；外部安全分析使用Slither静态分析器，确认生成合约审计结果与非LLM规则工具达79.4%一致。对9000份生成合约的系统分析揭示典型失败模式：逻辑遗漏占35.3%，状态转移错误占23.4%，复杂度引发性能退化。量化结果显示，生成合约综合得分比真实实现高8.29分，归因于LLM严格遵循规范的行为。所有数据、评估代码及生成合约均公开发布。
原文摘要 · Abstract (English)
We introduce SmartEval, a benchmark for systematically evaluating the quality of Solidity smart contracts generated by large language models (LLMs) from natural language specifications. SmartEval provides a corpus of 9,000 generated contracts paired with expert-written ground-truth implementations drawn from the FSMSCG dataset, a five-dimensional evaluation rubric covering functional completeness, variable fidelity, state-machine correctness, business-logic fidelity, and code quality, and a reproducible generation-and-evaluation pipeline. To validate the benchmark's reliability, we conduct three independent empirical studies: a five-condition ablation study (N=300 per condition) isolating the contribution of each pipeline component, a human expert evaluation by three Columbia University PhD researchers confirming automated scores align with expert judgment to within 0.34 points, and external security analysis via the Slither static analyzer confirming 79.4% agreement between the LLM auditor and a non-LLM rule-based tool. Systematic analysis of 9,000 generated contracts reveals characteristic failure modes (logic omissions at 35.3%, state transition errors at 23.4%, and complexity-driven degradation) and quantifies a +8.29 composite-score advantage of generated contracts over ground-truth implementations, attributable to LLMs' literal specification-following behavior. SmartEval establishes a reproducible, validated foundation for empirical research on LLM smart contract synthesis quality, with all data, evaluation code, and generated contracts publicly released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。