用验证器优先方法评估大模型生成云基础设施代码的实效，发现迭代优化可显著提升成功率。
Verifier-First Evaluation of Agentic LLMs for Infrastructure-as-Code Generation
- 先验证后生成，通过反馈循环逐步修正代码缺陷
- 迭代改进使GPT-4o成功率达84.4%，远超初始14.0%的基准
- 无需更新模型即可通过提示词优化提升生成质量，适合实际部署
从自然语言生成基础设施即代码（IaC）需满足平台架构、依赖关系与组织策略约束，而不仅是语法正确。我们对七种智能体策略在IaC-Eval v2（包含186个AWS/Terraform任务的现代化基准）上进行验证器优先的实证研究，该基准采用Rego v1意图策略。评估将失败细分为三个验证阶段（terraform validate、terraform plan、opa eval），并使用麦克奈马尔检验与威尔逊置信区间对所有成对比较（n=186，α=0.05）进行统计分析。报告五大发现：(1) 基于ReAct的主动检索（结合MCP或ChromaDB RAG）使Qwen2.5-Coder 7B的pass@1从14.0%提升至45.7%（p<0.0001），主要减少验证失败任务数（从144降至66）；(2) 带验证反馈的迭代优化实现62.9%（Qwen 7B）和84.4%（GPT-4o）的pass@1，表现出二元收敛——任务要么一次修复，要么耗尽重试预算；(3) GEPA反思式指令优化在仅80次验证引导推理下，使主动RAG基线提升7.5个百分点（p=0.026），证明提示优化可在不更新权重的情况下改善可验证IaC生成；(4) SIMBA无教师示范注入达到与主动RAG相当性能（p=1.0），但无法解决主导的SELF_DEFINED_PROPERTY错误类（占失败50%）；(5) 诊断性Rego注入实验显示，79%的后期OPA失败为信息缺口型，当策略文本可见时可解决（p=0.016），提示应增强策略可见性。
原文摘要 · Abstract (English)
Infrastructure-as-Code (IaC) generation from natural language requires satisfying provider schemas, dependency planning, and organizational policy constraints, not merely producing syntactically plausible configurations. We present a verifier-first empirical study of seven agentic strategies for Terraform generation evaluated on IaC-Eval v2, a modernized 186-task AWS/Terraform benchmark with Rego v1 intent policies. Our evaluation separates failures into three verifier stages (terraform validate, terraform plan, opa eval) and applies McNemar's test with Wilson confidence intervals on all pairwise comparisons (n=186, alpha=0.05). We report five principal findings. (1) Active retrieval via ReAct agents with MCP or ChromaDB-backed RAG raises Qwen2.5-Coder 7B from 14.0% to 45.7% pass@1 (p<0.0001), primarily by reducing VALIDATE_FAIL from 144 to 66 tasks. (2) Iterative refinement with verifier feedback achieves 62.9% (Qwen 7B) and 84.4% (GPT-4o) pass@1, exhibiting binary convergence -- tasks either resolve in one retry or exhaust the budget. (3) GEPA reflective instruction optimization raises the Active RAG baseline by +7.5 pp (p=0.026) using only 80 verifier-guided rollouts, providing evidence that prompt optimizers can improve verifiable IaC generation without weight updates. (4) SIMBA teacher-free demonstration injection achieves performance equivalent to Active RAG (p=1.0) without retrieval infrastructure, but fails to address the dominant SELF_DEFINED_PROPERTY error class (50% of failures). (5) A diagnostic Rego-injection experiment shows that 79% of post-refinement OPA failures are information-gap failures resolvable when policy text is visible (p=0.016), motivating policy
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。