用大模型检测以太坊智能合约漏洞，准确率超99%。
Detection Made Easy: Potentials of Large Language Models for Solidity Vulnerabilities
- 构建新数据集VulSmart，用大模型识别10类常见合约漏洞。
- 微调后闭源模型检测准确率达99%，类型识别94%，严重性判断98%。
- 思维链提示效果最佳，适合安全研究人员快速验证合约风险。
近年来，以太坊主网上大规模部署的Solidity智能合约吸引了大量利益驱动的攻击者。历史上著名的攻击事件包括2016年DAO攻击（损失5000万美元）、2017年Parity钱包漏洞（锁定1.46亿美元）、2018年Beautychain代币崩盘（市值从9亿美元归零）以及2022年NFT游戏链被盗（6亿美元以太币丢失）。本文系统研究了大语言模型（LLMs）在检测Solidity合约中OWASP Top Ten漏洞方面的潜力。我们提出一个新型、类别平衡、结构化且标注完整的数据集VulSmart，用于评估CodeLlama、Llama2、CodeT5、Falcon等开源模型，以及GPT-3.5 Turbo和GPT-4o Mini等闭源模型的性能。提出的SmartVD框架通过自动化与人工评估相结合的方式，在BLEU和ROUGE指标下进行测试，并探索零样本、少样本及思维链三种提示策略。结果表明，SmartVD优于多数开源模型，甚至超过部分闭源基线模型。微调后，GPT-3.5 Turbo与GPT-4o Mini分别实现99%的漏洞检测准确率、94%的类型识别率和98%的严重性判断准确率。其中，思维链提示对SmartVD表现最优，而微调后的闭源模型在零样本提示下表现更佳。
原文摘要 · Abstract (English)
The large-scale deployment of Solidity smart contracts on the Ethereum mainnet has increasingly attracted financially-motivated attackers in recent years. A few now-infamous attacks in Ethereum's history includes DAO attack in 2016 (50 million dollars lost), Parity Wallet hack in 2017 (146 million dollars locked), Beautychain's token BEC in 2018 (900 million dollars market value fell to 0), and NFT gaming blockchain breach in 2022 ($600 million in Ether stolen). This paper presents a comprehensive investigation of the use of large language models (LLMs) and their capabilities in detecting OWASP Top Ten vulnerabilities in Solidity. We introduce a novel, class-balanced, structured, and labeled dataset named VulSmart, which we use to benchmark and compare the performance of open-source LLMs such as CodeLlama, Llama2, CodeT5 and Falcon, alongside closed-source models like GPT-3.5 Turbo and GPT-4o Mini. Our proposed SmartVD framework is rigorously tested against these models through extensive automated and manual evaluations, utilizing BLEU and ROUGE metrics to assess the effectiveness of vulnerability detection in smart contracts. We also explore three distinct prompting strategies-zero-shot, few-shot, and chain-of-thought-to evaluate the multi-class classification and generative capabilities of the SmartVD framework. Our findings reveal that SmartVD outperforms its open-source counterparts and even exceeds the performance of closed-source base models like GPT-3.5 and GPT-4 Mini. After fine-tuning, the closed-source models, GPT-3.5 Turbo and GPT-4o Mini, achieved remarkable performance with 99% accuracy in detecting vulnerabilities, 94% in identifying their types, and 98% in determining severity. Notably, SmartVD performs best with the `chain-of-thought' prompting technique, whereas the fine-tuned closed-source models excel with the `zero-shot' prompting approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。