评测大模型生成云配置代码的安全部署能力，发现语法正确不等于安全合规。
Security-First Evaluation of Text-to-Terraform: Benchmarking LLMs and SLMs for Secure IaC Generation
- 在17种场景下测试7个模型，用Checkov和Trivy扫描生成的AWS Terraform代码。
- 顶级模型如Claude Opus 4仅23.1%通过安全检测，高语法通过率不保证安全合规。
- 提示工程无法确保安全，必须结合多工具自动扫描，适合关注云安全的开发者。
云配置错误仍是安全事件的主要原因，但大语言模型（LLMs）与小语言模型（SLMs）能否生成符合安全规范的基础设施即代码（IaC）仍存疑问。我们对七种模型进行了基准测试：三种闭源大模型（Claude Opus 4、GPT-5.4、Gemini 2.5 Pro）和四种开源小模型（Qwen2.5-Coder-14B、WizardCoder-33B、CodeLlama-13B、Magicoder-S-CL-7B），在17个AWS Terraform生成场景中评估其表现，将Checkov与Trivy扫描器集成至GitLab CI/CD流水线，并在三个安全等级下测试两种提示策略（pass@5）。结果表明，语法有效性与安全合规性在生成的IaC中几乎无关：某模型语法通过率达77.8%，但Checkov合规率为零；而Claude Opus 4在详细提示下达到23.1% Checkov通过率与92.5% Trivy通过率。因此，仅靠提示工程不足以为继，无论模型类型或提示策略如何，自动化多工具扫描仍是辅助生成IaC所必需的环节。所有实验资产均已公开。
原文摘要 · Abstract (English)
Cloud misconfiguration remains a leading cause of security incidents, yet whether LLMs and SLMs can generate security-compliant Infrastructure-as-Code is an open question. We benchmark seven models, three closed LLMs (Claude Opus 4, GPT-5.4, Gemini 2.5 Pro) and four open SLMs (Qwen2.5-Coder-14B, WizardCoder-33B, CodeLlama-13B, Magicoder-S-CL-7B), on AWS Terraform generation across 17 scenarios, integrating Checkov and Trivy scanners into a GitLab CI/CD pipeline and evaluating two prompt strategies at three security levels (pass@5). Syntactic validity and security compliance are largely orthogonal properties in LLM-generated IaC, a model that reliably produces well-formed Terraform does not necessarily produce secure Terraform: WizardCoder-33B achieves 77.8% validate rate yet zero Checkov compliance, while Claude Opus 4 reaches 23.1% Checkov and 92.5% Trivy pass rates under detailed security prompting. Consequently, prompt engineering alone is insufficient: automated multi-tool scanning remains a necessary complement to LLM-assisted IaC generation regardless of model family or prompt strategy. All artifacts are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。