用提示工程提升大模型生成代码的安全性，实测效果显著
Benchmarking Prompt Engineering Techniques for Secure Code Generation with GPT Models
- 设计自动化基准测试，评估多种提示策略对代码安全的影响
- 针对GPT-4o等模型，安全提示前缀可降低56%漏洞率
- 迭代提示能修复41.9%~68.7%已有漏洞，适合实际开发集成
提示工程可减少大语言模型（LLMs）的推理错误，但其在缓解大模型生成代码漏洞方面的效果仍不明确。为填补这一空白，我们构建了一个自动化基准测试，用于评估不同提示工程策略对代码安全的影响。该基准使用两个经过同行评审的提示数据集，并采用静态扫描器大规模评估代码安全性。我们在GPT-3.5-turbo、GPT-4o和GPT-4o-mini上测试了多种提示工程方法。结果显示，对于GPT-4o和GPT-4o-mini，安全导向的提示前缀可将漏洞发生率降低高达56%。此外，所有测试模型在使用迭代提示时，均能检测并修复41.9%至68.7%的已有漏洞。最后，我们提出一个‘提示代理’（prompt agent），展示了最有效技术如何融入真实开发流程。
原文摘要 · Abstract (English)
Prompt engineering reduces reasoning mistakes in Large Language Models (LLMs). However, its effectiveness in mitigating vulnerabilities in LLM-generated code remains underexplored. To address this gap, we implemented a benchmark to automatically assess the impact of various prompt engineering strategies on code security. Our benchmark leverages two peer-reviewed prompt datasets and employs static scanners to evaluate code security at scale. We tested multiple prompt engineering techniques on GPT-3.5-turbo, GPT-4o, and GPT-4o-mini. Our results show that for GPT-4o and GPT-4o-mini, a security-focused prompt prefix can reduce the occurrence of security vulnerabilities by up to 56%. Additionally, all tested models demonstrated the ability to detect and repair between 41.9% and 68.7% of vulnerabilities in previously generated code when using iterative prompting techniques. Finally, we introduce a "prompt agent" that demonstrates how the most effective techniques can be applied in real-world development workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。