arXiv:2509.05303cs.DCcs.AI2025-09被引 6

首个跨云IaC格式的智能生成评测基准,助力AI自动管理多平台基础设施。

Multi-IaC-Eval: Benchmarking Cloud Infrastructure as Code Across Multiple Formats

  • 构建多格式IaC合成数据集,涵盖AWS CloudFormation、Terraform与CDK。
  • LLM在语法正确性上表现优异(>95%成功率),但语义对齐仍存挑战。
  • 适合研究AI辅助云架构、自动化运维及提示工程的开发者和研究人员。

基础设施即代码(IaC)是现代云计算的核心,通过可机器读取的配置文件定义和管理基础设施。然而,不同云服务商采用不同的IaC格式,缺乏统一标准使云架构师需掌握多种语言,增加了部署复杂性。尽管大语言模型(LLMs)在自动化IaC创建与维护方面展现潜力,但进展受限于跨多格式的全面评估基准缺失。本文提出Multi-IaC-Bench,一个用于评估基于LLM的IaC生成与修改的新型基准数据集,覆盖AWS CloudFormation、Terraform和Cloud Development Kit(CDK)格式。数据集包含三元组:初始IaC模板、自然语言修改请求及对应更新后的模板,由合成数据生成管道生成并经严格验证。我们在Multi-IaC-Bench上评估多个先进LLMs,结果显示现代模型在跨格式生成语法正确的IaC方面成功率超过95%,但在语义对齐与处理复杂基础设施模式方面仍面临显著挑战。消融实验表明提示工程与重试机制对成功生成至关重要。我们公开发布Multi-IaC-Bench,以推动人工智能辅助基础设施管理的研究,并建立该关键领域的标准化评估指标。

原文摘要 · Abstract (English)

Infrastructure as Code (IaC) is fundamental to modern cloud computing, enabling teams to define and manage infrastructure through machine-readable configuration files. However, different cloud service providers utilize diverse IaC formats. The lack of a standardized format requires cloud architects to be proficient in multiple IaC languages, adding complexity to cloud deployment. While Large Language Models (LLMs) show promise in automating IaC creation and maintenance, progress has been limited by the lack of comprehensive benchmarks across multiple IaC formats. We present Multi-IaC-Bench, a novel benchmark dataset for evaluating LLM-based IaC generation and mutation across AWS CloudFormation, Terraform, and Cloud Development Kit (CDK) formats. The dataset consists of triplets containing initial IaC templates, natural language modification requests, and corresponding updated templates, created through a synthetic data generation pipeline with rigorous validation. We evaluate several state-of-the-art LLMs on Multi-IaC-Bench, demonstrating that while modern LLMs can achieve high success rates (>95%) in generating syntactically valid IaC across formats, significant challenges remain in semantic alignment and handling complex infrastructure patterns. Our ablation studies highlight the importance of prompt engineering and retry mechanisms in successful IaC generation. We release Multi-IaC-Bench to facilitate further research in AI-assisted infrastructure management and establish standardized evaluation metrics for this crucial domain.

基础设施即代码大模型应用云原生评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。