用AI生成云配置文件,通过反复试错提升部署成功率。
Deployability-Centric Infrastructure-as-Code Generation: Fail, Learn, Refine, and Succeed through LLM-Empowered DevOps Simulation

- 构建迭代反馈机制,结合格式校验与真实部署测试
- 10轮内使54.6%~91.6%的配置可成功部署
- 人工干预可让成功率超90%,适合想自动化运维的开发者
基础设施即代码(IaC)生成有望自动化云资源部署。尽管大语言模型(LLM)能从自然语言生成IaC模板,但现有评估仅关注语法正确性,忽视实际部署可行性。六款主流LLM首次部署成功率仅为20.8%~30.2%。本文构建首个以部署能力为核心的IaC评测基准DPIaC-Eval,涵盖153个真实场景、58种服务。提出基于LLM的部署导向框架IaCGen,通过格式验证、语法检查和实时部署的迭代反馈机制,模拟真实开发运维流程。结果表明,IaCGen可在10次迭代内将所有评估模型生成的IaC模板部署成功率提升至54.6%~91.6%。引入人工反馈后,所有模型在25轮内通过率超过90%。此外,生成配置在用户意图契合度(覆盖率25.2%)和安全合规性(合规率8.4%)方面表现不佳,凸显该领域仍需深入研究。
原文摘要 · Abstract (English)
Infrastructure-as-Code (IaC) generation holds significant promise for automating cloud infrastructure provisioning. Recent advances in Large Language Models (LLMs) present a promising opportunity to democratize IaC development by generating deployable infrastructure templates from natural language descriptions. However, current evaluation focuses on syntactic correctness while ignoring deployability, the critical measure of the utility of IaC configuration files. Six state-of-the-art LLMs performed poorly on deployability, achieving only 20.8$\sim$30.2% deployment success rate on the first attempt. In this paper, we construct DPIaC-Eval, the first deployability-centric IaC template benchmark consisting of 153 real-world scenarios cross 58 unique services. Also, we propose an LLM-based deployability-centric framework, dubbed IaCGen, that uses iterative feedback mechanism encompassing format verification, syntax checking, and live deployment stages, thereby closely mirroring the real DevOps workflows. Results show that IaCGen can make 54.6$\sim$91.6% generated IaC templates from all evaluated models deployable in the first 10 iterations. Additionally, human-in-the-loop feedback that provide direct guidance for the deployability errors, can further boost the performance to over 90% passItr@25 on all evaluated LLMs. Furthermore, we explore the trustworthiness of the generated IaC templates on user intent alignment and security compliance. The poor performance (25.2% user requirement coverage and 8.4% security compliance rate) indicates a critical need for continued research in this domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。