本地运行大模型可检测43%-45%的Python漏洞，但定位不准。
An Empirical Evaluation of Locally Deployed LLMs for Bug Detection in Python Code
- 在本地部署LLaMA 3.2和Mistral模型，零样本检测函数级漏洞。
- 在BugsInPy上检测349个漏洞，准确率43%-45%，多数仅定位问题区域。
- 适合隐私敏感或资源受限场景，但复杂代码定位仍困难。
大型语言模型（LLMs）在代码生成与分析等软件工程任务中表现优异，但多数研究依赖云端模型或专用硬件，限制了其在隐私敏感或资源受限环境中的应用。本文针对真实场景下的Python漏洞检测，系统评估了两个本地部署的LLM——LLaMA 3.2和Mistral。基于BugsInPy基准，我们在17个项目的349个漏洞上，采用函数级零样本提示策略，并通过自动化关键词评估框架进行测试。结果表明，本地模型的准确率在43%至45%之间，虽能识别出问题代码区域，但难以精确定位修复方案。不同项目间性能差异显著，凸显代码库特征的影响。研究显示，本地模型可有效识别部分漏洞，但在处理复杂、依赖上下文的真实开发场景时，精准定位仍具挑战。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated strong performance on a wide range of software engineering tasks, including code generation and analysis. However, most prior work relies on cloud-based models or specialized hardware, limiting practical applicability in privacy-sensitive or resource-constrained environments. In this paper, we present a systematic empirical evaluation of two locally deployed LLMs, LLaMA 3.2 and Mistral, for real-world Python bug detection using the BugsInPy benchmark. We evaluate 349 bugs across 17 projects using a zero-shot prompting approach at the function level and an automated keyword-based evaluation framework. Our results show that locally executed models achieve accuracy between 43% and 45%, while producing a large proportion of partially correct responses that identify problematic code regions without pinpointing the exact fix. Performance varies significantly across projects, highlighting the importance of codebase characteristics. The results demonstrate that local models can identify a meaningful share of bugs, though precise localization remains difficult for locally executed LLMs, particularly when handling complex and context dependent bugs in realistic development scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。