评测大模型零样本推理在以太坊合约漏洞检测中的表现
Benchmarking Zero-Shot Reasoning Approaches for Error Detection in Solidity Smart Contracts
- 用零样本提示策略测试主流大模型对合约漏洞的识别能力
- 思维链和树状思维使召回率升至95%~99%，但准确率下降
- Claude 3 Opus在分类任务中得分最高，达90.8分
智能合约在区块链系统中承担核心的金融与运营逻辑，但其易受细微安全缺陷影响，可能引发重大经济损失并损害信任。大语言模型为自动化漏洞检测带来新机遇，但不同提示策略与模型选择在真实场景下的效果仍不明确。本文基于400个合约的平衡数据集，在两个任务上评估前沿大模型：(i) 漏洞检测，模型进行二分类判断合约是否存在漏洞；(ii) 漏洞分类，模型需将问题归入具体漏洞类别。采用零样本提示策略，包括零样本、零样本思维链（CoT）和零样本树状思维（ToT）。在漏洞检测任务中，CoT与ToT显著提升召回率（常达95%~99%），但通常降低精度，表明决策更敏感且误报增多。在漏洞分类任务中，Claude 3 Opus在ToT提示下取得最高加权F1分数（90.8），紧随其后的是其CoT版本。
原文摘要 · Abstract (English)
Smart contracts play a central role in blockchain systems by encoding financial and operational logic. Still, their susceptibility to subtle security flaws poses significant risks of financial loss and erosion of trust. LLMs create new opportunities for automating vulnerability detection, yet the effectiveness of different prompting strategies and model choices in real-world contexts remains uncertain. This paper evaluates state-of-the-art LLMs on Solidity smart contract analysis using a balanced dataset of 400 contracts under two tasks: (i) Error Detection, where the model performs binary classification to decide whether a contract is vulnerable, and (ii) Error Classification, where the model must assign the predicted issue to a specific vulnerability category. Models are evaluated using zero-shot prompting strategies, including zero-shot, zero-shot Chain-of-Thought (CoT), and zero-shot Tree-of-Thought (ToT). In the Error Detection task, CoT and ToT substantially increase recall (often approaching ~ 95--99%), but typically reduce precision, indicating a more sensitive decision regime with more false positives. In the Error Classification task, Claude 3 Opus attains the best Weighted F1-score (90.8) under the ToT prompt, followed closely by its CoT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。