测试大模型在智能合约安全检测中的表现，发现其误报率高且依赖命名习惯。
Benchmarking LLM-Based Static Analysis for Secure Smart Contract Development: Reliability, Limitations, and Potential Hybrid Solutions
- 用自研框架评估大模型输出，发现其依赖命名等非语义特征
- 分类准确率达92%，但存在严重误报问题
- 适合做传统静态分析的补充，不宜独立替代人工审计
区块链交易不可逆，识别智能合约漏洞对保障系统安全至关重要。尽管大型语言模型（LLMs）已融入开发流程，但其作为自主安全审计工具的可靠性尚未验证。本文评估当前生成式模型是否可替代或仅补充传统静态分析工具。结果表明，LLM效能受内在词汇偏差及外部数据输入缺乏严格验证影响，过度依赖标识符命名等非语义启发式规则，导致误报频发。同时，提示工程揭示精确率与召回率间的权衡。研究基于自研自动化框架，实现模型输出分类92%准确率。
原文摘要 · Abstract (English)
The irreversible nature of blockchain transactions makes the identification of smart contract vulnerabilities an essential requirement for secure system development. While Large Language Models (LLMs) are increasingly integrated into developer workflows, their reliability as autonomous security auditors remains unproven. We assess whether current generative models are a viable replacement for, or only a complement to, traditional static-analysis tools. Our findings indicate that LLM efficacy is undermined by both inherent lexical bias and a lack of rigorous validation of external data inputs. This reliance on non-semantic heuristics, such as identifier naming, leads to a high frequency of false positives. Furthermore, prompting techniques reveal a trade-off between precision and recall. These results were derived using our custom automated framework, which achieves 92% accuracy in classifying model outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。