构建真实IT自动化任务评估框架，揭示大模型在运维安全等场景的实战短板。
ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks
- 设计可一键运行的基准测试流程，覆盖SRE、CISO、FinOps三类核心场景
- 实测顶尖模型仅解决13.8%运维任务、25.2%安全任务，金融运维零成功率
- 开源94个真实场景，支持社区扩展，适合研究可信AI自动化者参考
实现用AI代理自动化关键IT任务，依赖于对解决方案有效性的衡量与理解。本文提出ITBench，一种系统化评估AI代理在真实世界IT自动化任务中表现的框架。初始版本聚焦三大领域：站点可靠性工程（SRE）、合规与安全运营（CISO）及财务运营（FinOps）。该框架提供开箱即用的工作流和可解释指标，帮助研究人员把握AI代理在IT自动化中的挑战与机遇。ITBench包含初始94个真实世界任务场景，支持社区持续扩展。实验结果表明，基于当前最先进模型的代理仅能解决13.8%的SRE任务、25.2%的CISO任务，而对所有FinOps任务均无法完成。我们期望ITBench能成为推动可靠、安全、快速的AI驱动式IT自动化的关键工具。
原文摘要 · Abstract (English)
Realizing the vision of using AI agents to automate critical IT tasks depends on the ability to measure and understand effectiveness of proposed solutions. We introduce ITBench, a framework that offers a systematic methodology for benchmarking AI agents to address real-world IT automation tasks. Our initial release targets three key areas: Site Reliability Engineering (SRE), Compliance and Security Operations (CISO), and Financial Operations (FinOps). The design enables AI researchers to understand the challenges and opportunities of AI agents for IT automation with push-button workflows and interpretable metrics. ITBench includes an initial set of 94 real-world scenarios, which can be easily extended by community contributions. Our results show that agents powered by state-of-the-art models resolve only 13.8% of SRE scenarios, 25.2% of CISO scenarios, and 0% of FinOps scenarios. We expect ITBench to be a key enabler of AI-driven IT automation that is correct, safe, and fast.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。