测试大模型在真实IT自动化任务中的表现，发现效果远未达标。
Large Language Models for IT Automation Tasks: Are We There Yet?
- 构建126个真实任务的基准，要求模型生成可执行的Ansible脚本。
- 14个开源模型平均通过率不足12%，多数失败因状态追踪错误。
- 揭示大模型在状态管理与模块知识上的短板，适合关注自动化落地的研究者。
大语言模型在代码生成方面展现潜力，但在实际IT自动化任务（如Ansible）中的有效性仍缺乏研究。现有基准多依赖合成任务，无法反映从业者的真实需求。本文提出ITAB（IT自动化任务基准），包含126项多样化任务（如服务器配置、文件管理），每项任务均需考虑状态一致性——这是IT自动化工具的独特属性。ITAB通过在受控环境中动态执行，评估大模型生成功能性Ansible脚本的能力。我们测试了14个开源大模型，其在pass@10指标下的最高通过率仅为12%。分析1,411次执行失败后发现，主要问题集中在两类语义错误：状态一致性相关推理失败（44.87%，包括变量、主机、路径、模板问题）和模块特异性执行知识缺失（24.37%，含属性/参数和模块调用错误）。结果表明，当前开源大模型在状态追踪与领域专用执行理解上存在显著局限，实现可靠的IT自动化仍需重大改进。
原文摘要 · Abstract (English)
LLMs show promise in code generation, yet their effectiveness for IT automation tasks, particularly for tools like Ansible, remains understudied. Existing benchmarks rely primarily on synthetic tasks that fail to capture the needs of practitioners who use IT automation tools, such as Ansible. We present ITAB (IT Automation Task Benchmark), a benchmark of 126 diverse tasks (e.g., configuring servers, managing files) where each task accounts for state reconciliation: a property unique to IT automation tools. ITAB evaluates LLMs' ability to generate functional Ansible automation scripts via dynamic execution in controlled environments. We evaluate 14 open-source LLMs, none of which accomplish pass@10 at a rate beyond 12%. To explain these low scores, we analyze 1,411 execution failures across the evaluated LLMs and identify two main categories of prevalent semantic errors: failures in state reconciliation related reasoning (44.87% combined from variable (11.43%), host (11.84%), path(11.63%), and template (9.97%) issues) and deficiencies in module-specific execution knowledge (24.37% combined from Attribute and parameter (14.44%) and module (9.93%) errors). Our findings reveal key limitations in open-source LLMs' ability to track state changes and apply specialized module knowledge, indicating that reliable IT automation will require major advances in state reasoning and domain-specific execution understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。