开源大模型代理无法替代专业代码安全检测工具
Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment
- 用三个开源大模型构建智能代理进行代码安全扫描
- 代理误报率高，检测准确率远低于专业工具Bandit
- 适合对安全检测原理感兴趣的开发者了解局限性
本文评估了基于通用开源大语言模型的智能体在网络安全领域的实际效能。研究使用Ollama托管的三种主流开源模型，构建通用型生成式AI代理，通过精确率、召回率、误报数量及综合评分等指标，与经过验证的静态应用安全测试（SAST）工具Bandit进行对比。结果表明，在真实应用场景下，当前开源GenAI大模型代理尚不具备胜任专业代码安全扫描任务的能力，其性能显著低于成熟SAST工具。研究反驳了‘通用大模型代理可替代传统安全检测工具’这一流行观点。
原文摘要 · Abstract (English)
This paper explores the value of agentic AI tools for cybersecurity purposes. We evaluate the efficacy of a general-purpose GenAI Large Language Model- (GenAI-) based agent when powered by three different Ollama-hosted general-purpose open source models. We assess each agent's performance using precision, recall, false positive count, and a calculated composite score based upon the interplay of the captured metrics, against the baseline performance of an existing, vetted Static Application Security Testing (SAST) tool, Bandit. Our findings refute the notion that a modern open-source GenAI LLM-based agent is currently suitable for the specialized task of SAST scanning under realistic conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。