用反思提示提升大模型生成代码的安全性，三轮后准确率从70.74%升至79.43%
Secure Code Generation at Scale with Reflexion
- 采用三轮反思提示，逐步优化代码安全性
- 首轮不安全率25%-33%，三轮后平均准确率达79.43%
- 适合关注代码安全的大模型应用开发者
大型语言模型广泛用于代码编写与重构，但运行正确的代码未必安全。本文使用Instruct Prime评估五种指令微调的代码大模型，采用零样本基线（t0）和三轮反思提示（t3）方法，以不安全代码检测器（ICD）衡量安全性，结果按修复率（Repair）、回归率（Regression）和净收益（NetGain）指标报告，考虑编程语言与弱安全漏洞类别（CWE）。结果显示，零样本下约25%-33%的程序存在不安全问题，其中弱加密与配置相关漏洞最难避免，而模板化漏洞如XSS、代码注入和硬编码密钥处理较可靠。Python安全率最高，C和C#最低，其余语言居中。反思提示使所有模型安全性提升，平均准确率从t0的70.74%增至t3的79.43%，首两轮收益最大，后续递减。复现包已公开于https://doi.org/10.5281/zenodo.17065846。
原文摘要 · Abstract (English)
Large language models (LLMs) are now widely used to draft and refactor code, but code that works is not necessarily secure. We evaluate secure code generation using the Instruct Prime, which eliminated compliance-required prompts and cue contamination, and evaluate five instruction-tuned code LLMs using a zero-shot baseline and a three-round reflexion prompting approach. Security is measured using the Insecure Code Detector (ICD), and results are reported by measuring Repair, Regression, and NetGain metrics, considering the programming language and CWE family. Our findings show that insecurity remains common at the first round: roughly 25-33% of programs are insecure at a zero-shot baseline (t0 ). Weak cryptography/config-dependent bugs are the hardest to avoid while templated ones like XSS, code injection, and hard-coded secrets are handled more reliably. Python yields the highest secure rates; C and C# are the lowest, with Java, JS, PHP, and C++ in the middle. Reflexion prompting improves security for all models, improving average accuracy from 70.74% at t0 to 79.43% at t3 , with the largest gains in the first round followed by diminishing returns. The trends with Repair, Regression, and NetGain metrics show that applying one to two rounds produces most of the benefits. A replication package is available at https://doi.org/10.5281/zenodo.17065846.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。