首次对比大模型与人类生成的代码安全，发现模型漏洞密度更高。
Compared to What? A Human-Anchored Security Benchmark for LLM-Generated Infrastructure-as-Code
- 构建100个分层场景基准，对比12种模型与634份人工代码的安全性
- 模型漏洞密度是人工的3.21至4.9倍，尤其在简单任务中更差
- 厂商增强推理显著优于提示链式思考，但仅用不到1%输出预算
大型语言模型越来越多地用于生成基础设施即代码（IaC），而单一不安全默认配置可直接部署至生产环境。以往评估仅报告模型生成IaC的漏洞数量，但缺乏人类基线，无法判断模型是否比工程师更差。我们提出GenIaC-SecBench，一个包含100个按架构复杂度分层的部署场景的基准，覆盖来自四个供应商的12种模型配置，生成1,196份IaC文件，并由三个独立策略引擎（Checkov、Trivy、KICS）扫描。关键的是，我们还使用相同工具链扫描了634份人工编写的IaC模板，提供了首个规模匹配的人类安全基线。漏洞密度与文件大小呈强负相关（Spearman ρ = -0.55，p < 10⁻⁷⁷），意味着未匹配大小的比较实质上衡量的是规模而非安全性。在声明资源数匹配的情况下，所有模型配置的漏洞密度均在人类的3.21至3.87倍之间，且在简单任务中差距更大（单资源时达4.9倍，二十个以上资源时为1.4倍）。我们分解了生成方式：标准生成、提示工程链式思考、厂商扩展思维API。厂商扩展思维显著优于提示链式思考（-12.0%，p = 0.0013），而提示链式思考与标准生成无差异（-1.3%，非显著）。令牌分析显示，扩展思维仅使用不到1%的输出预算，解释其效果有限。两个负面结果亦浮现：可部署性与漏洞无相关性（r = 0.158，p = 0.625），且经典完整案例弗里德曼检验在现实基准设计中不可行，因此提出采用Skillings-Mack统计量。所有代码、数据与重生成脚本均已公开。
原文摘要 · Abstract (English)
Large language models are increasingly used to author Infrastructure-as-Code (IaC), where a single insecure default can be deployed directly into production. Prior evaluations report raw vulnerability counts for model-generated IaC, but without a human baseline they cannot determine whether models are actually worse than engineers. We introduce GenIaC-SecBench, a benchmark of 100 deployment scenarios stratified by architectural complexity, evaluated across 12 model configurations from four vendors, producing 1,196 IaC artifacts scanned by three independent policy engines (Checkov, Trivy, KICS). Critically, we also scan 634 human-authored IaC templates with the same toolchain, providing the first size-matched human security baseline. Vulnerability density is strongly inverse to artifact size (Spearman $ρ= -0.55$, $p < 10^{-77}$), meaning unmatched comparisons measure size rather than security. When matched on declared-resource count, all model configurations fall within 3.21x--3.87x the human vulnerability density, with the gap widening for simpler tasks (4.9x at one resource, 1.4x at twenty or more). We decompose reasoning into standard generation, prompt-engineered chain-of-thought, and vendor extended-thinking APIs. Vendor extended thinking significantly outperforms prompted chain-of-thought ($-12.0\%$, $p = 0.0013$), while prompted chain-of-thought is indistinguishable from standard generation ($-1.3\%$, n.s.). Token instrumentation shows extended thinking uses under 1\% of the output budget, explaining its bounded effect. Two negative results also emerge: deployability does not correlate with vulnerability ($r = 0.158$, $p = 0.625$), and classical complete-case Friedman testing is infeasible for realistic benchmark designs, motivating the Skillings-Mack statistic. All code, data, and regeneration scripts are released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。