五款主流大模型生成代码普遍存在缺陷,功能达标不等于安全可用。
Assessing the Quality and Security of AI-Generated Code: A Quantitative Analysis
- 用SonarQube对4442份Java代码做静态分析,检测缺陷。
- 所有模型都存在严重漏洞,如硬编码密码和路径遍历风险。
- 功能通过率高不代表代码质量好,需额外安全验证。
本研究对五款主流大语言模型(Claude Sonnet 4、Claude 3.7 Sonnet、GPT-4o、Llama 3.2 90B、OpenCoder 8B)生成的代码进行了定量评估。基于4,442个Java编程任务,采用SonarQube进行全面静态分析,结果表明:尽管这些模型能生成功能性代码,但普遍存在各类软件缺陷,包括漏洞、安全风险与代码异味。这些缺陷并非孤立出现,可能源于当前大模型代码生成方法的系统性局限。特别地,硬编码密码与路径遍历等关键严重问题在多个模型中均被发现。研究未发现功能表现(以单元测试通过率Pass@1衡量)与代码整体质量及安全性(以通过测试代码中的SonarQube问题数衡量)之间存在直接关联,说明功能达标不能反映真实代码质量。研究目标并非排名,而是揭示所有模型均存在共性弱点,强调静态分析是识别潜在缺陷的重要手段,对组织在软件开发中部署AI具有重要警示意义。
原文摘要 · Abstract (English)
This study presents a quantitative evaluation of the code quality and security of five prominent Large Language Models (LLMs): Claude Sonnet 4, Claude 3.7 Sonnet, GPT-4o, Llama 3.2 90B, and OpenCoder 8B. While prior research has assessed the functional performance of LLM-generated code, this research tested LLM output from 4,442 Java coding assignments through comprehensive static analysis using SonarQube. The findings suggest that although LLMs can generate functional code, they also introduce a range of software defects, including bugs, security vulnerabilities, and code smells. These defects do not appear to be isolated; rather, they may represent shared weaknesses stemming from systemic limitations within current LLM code generation methods. In particular, critically severe issues, such as hard-coded passwords and path traversal vulnerabilities, were observed across multiple models. These results indicate that LLM-generated code requires verification in order to be considered production-ready. This study found no direct correlation between a model's functional performance (measured by Pass@1 rate of unit tests) and the overall quality and security of its generated code, measured by the number of SonarQube issues in benchmark solutions that passed the functional tests. This suggests that functional benchmark performance score is not a good indicator of overall code quality and security. The goal of this study is not to rank LLM performance but to highlight that all evaluated models appear to share certain weaknesses. Consequently, these findings support the view that static analysis can be a valuable instrument for detecting latent defects and an important safeguard for organizations that deploy AI in software development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。