对比五种提示法生成代码的安全性，发现改提示难降漏洞总数
An Empirical Evaluation of LLM-Generated Code Security Across Prompting Methods
- 用安全漏洞映射增强提示，引导模型思考安全问题
- 不同提示法对漏洞类型分布有影响，但整体漏洞数未显著减少
- 代码语言差异使提示效果不同，需定制化设计
大型语言模型（LLMs）在自动化代码生成中的应用提升了开发效率，但也常带来安全缺陷，如弱加密和输入验证不当。本文对五种LLM及四种编程语言（Java、C++、C、Python）生成的代码进行系统性实证评估，考察多种提示工程方法的影响。提出一种基于弱点映射（CWE）的零样本思维链（WA-0CoT）提示策略，以引入安全上下文。基于卡方检验的分析显示，各提示方法间漏洞频率与密度无统计学显著差异。然而，提示策略系统性地改变了漏洞类别组成，且影响随编程语言变化。结果表明，尽管安全提示能改变漏洞结构，但无法可靠降低总体漏洞水平。研究强调在评估代码安全性时，应考虑语言特性和模型特性进行提示设计。
原文摘要 · Abstract (English)
The growing use of Large Language Models (LLMs) for automated code generation has enhanced software development efficiency, but often at the cost of security. Generated code frequently overlooks critical concerns, leaving it vulnerable to issues such as weak encryption and improper input validation. To investigate this problem, we present a comprehensive empirical evaluation of the security quality of LLM-generated code across five LLMs and four programming languages (Java, C++, C, and Python), examining the impact of multiple prompt engineering methods. We introduce a weaknesses-aware zero-shot chain-of-thought (WA-0CoT) prompting strategy that enriches prompts with security context using CWE mappings to guide model reasoning. Our empirical analysis, supported by chi-square tests, finds no statistically significant reductions in vulnerability frequency or density across prompt methods. However, prompting strategies, including WA-0CoT, systematically influence the compositional distribution of CWE categories, with effects varying by programming language. These findings suggest that while security-aware prompting alters the structure of generated weaknesses, prompt engineering alone is insufficient to reliably reduce overall vulnerability levels. The results highlight the importance of language-aware and model-aware prompt design when evaluating the security properties of LLM-generated code.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。