用结构化配置知识提升大模型生成正确云代码的能力
IaC Generation with LLMs: An Error Taxonomy and A Study on Configuration Knowledge Injection
- 通过图结构知识注入改进大模型的IaC生成
- 技术验证成功率从27.1%提升至75.3%
- 适合关注云基础设施自动化与大模型应用的研究者
当前大语言模型在生成正确且符合意图的基础设施即代码（IaC）方面表现不佳。本研究针对Terraform场景，通过系统性注入结构化配置知识来改进基于LLM的IaC生成。为此，对现有IaC-Eval基准进行了增强，引入云环境模拟和自动错误分析，并提出了一种新的LLM辅助IaC生成错误分类体系。实验对比了从基础检索增强生成（RAG）到更复杂的图RAG方法，包括对图组件的语义增强及资源间依赖关系建模。结果表明，尽管基线模型整体成功率为27.1%，但注入结构化知识后，技术验证成功率提升至75.3%，整体成功率提升至62.6%。然而，意图对齐性能趋于停滞，揭示出‘正确性-一致性鸿沟’：模型可成为熟练编码者，却难以胜任复杂意图理解的架构设计任务。
原文摘要 · Abstract (English)
Large Language Models (LLMs) currently exhibit low success rates in generating correct and intent-aligned Infrastructure as Code (IaC). This research investigated methods to improve LLM-based IaC generation, specifically for Terraform, by systematically injecting structured configuration knowledge. To facilitate this, an existing IaC-Eval benchmark was significantly enhanced with cloud emulation and automated error analysis. Additionally, a novel error taxonomy for LLM-assisted IaC code generation was developed. A series of knowledge injection techniques was implemented and evaluated, progressing from Naive Retrieval-Augmented Generation (RAG) to more sophisticated Graph RAG approaches. These included semantic enrichment of graph components and modeling inter-resource dependencies. Experimental results demonstrated that while baseline LLM performance was poor (27.1% overall success), injecting structured configuration knowledge increased technical validation success to 75.3% and overall success to 62.6%. Despite these gains in technical correctness, intent alignment plateaued, revealing a "Correctness-Congruence Gap" where LLMs can become proficient "coders" but remain limited "architects" in fulfilling nuanced user intent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。