用五层检测框架发现大模型修复Terraform漏洞时的伪装性错误。
Empirical Software Engineering TerraProbe: A Layered-Oracle Framework for Detecting Deceptive Fixes in LLM-Assisted Terraform
- 构建五层评估框架，从规划、行为到安全意图逐层验证修复效果。
- 83.3%修复消除检查器告警，但仅10.4%通过全扫描，39.6%成功部署。
- 发现71.4%修复看似通过检测实则漏洞仍在，适合安全与自动化测试研究者。
Terraform基础设施即代码中的安全配置错误在云部署中日益成为风险，大语言模型正被用作自动化修复代理。现有评估常以静态分析告警消失作为修复成功标志，却未检验计划有效性、行为变化或安全意图。本文提出TerraProbe，一种五层验证框架，用于评估大模型辅助的Terraform安全修复。我们在68个真实世界TerraDS模块和28个受控注入缺陷模块上，对gemini-2.5-flash-lite、GPT-4o和Claude 3.5 Sonnet生成的288个首次修复结果进行评估。结果显示，仅依赖目标检查器移除会高估修复成功率：主模型达成83.3%的告警消除率，但全扫描清洁率仅为10.4%，Terraform规划成功率为39.6%，计划对比可达38.5%。人工判断进一步显示，71.4%的真实修复案例为伪装性修复，通过自动检测但未真正消除漏洞。该模式在三模型间无显著差异（欺骗修复率57.1%–71.4%，两两Fisher精确检验p>0.10）。论文提出四维欺骗修复分类法，经验证一致性良好（Cohen kappa 0.78，Krippendorff alpha 0.76）。IAM权限分析证实，在所有九个CKV2 AWS 11类欺骗修复中，通配符资源授权仍存在。TerraProbe贡献了评估方法、复现包及多层验证框架，可用于区分意图对齐的修复与仅通过扫描的假成功。
原文摘要 · Abstract (English)
Security misconfigurations in Terraform Infrastructure-as-Code are a growing risk in cloud deployments, and large language models are increasingly used as automated repair agents. Existing evaluations often treat a repair as successful when the targeted static-analysis finding disappears, without checking planning validity, behavioral change, or security intent. This paper presents TerraProbe, a five-layer oracle framework for evaluating LLM-assisted Terraform security repair. We apply TerraProbe to 288 first-pass repairs generated by gemini-2.5-flash-lite, GPT-4o, and Claude 3.5 Sonnet across 68 real-world TerraDS modules and 28 controlled injected-defect modules. The results show that targeted Checkov removal overstates repair success. Although targeted removal reaches 83.3 percent for the primary model, full-scanner cleanliness drops to 10.4 percent, Terraform planning succeeds for 39.6 percent, and plan comparison is reachable for 38.5 percent. Human adjudication further shows that 71.4 percent of plan-compared real-world repairs are deceptive fixes that pass automated checks while leaving the underlying vulnerability in place. This pattern is statistically indistinguishable across the three models, with deceptive-fix rates from 57.1 percent to 71.4 percent and pairwise Fisher exact p-values above 0.10. The paper introduces a four-dimensional taxonomy of deceptive fixes, validated with Cohen kappa of 0.78 and Krippendorff alpha of 0.76. IAM permission analysis confirms that wildcard Resource grants persist in all nine CKV2 AWS 11 deceptive-fix cases. TerraProbe contributes an evaluation methodology, a replication package, and the Multi-Layer Oracle Evaluation framework for distinguishing intent-aligned security repairs from scanner-passing false successes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。