首个面向微服务自愈的LLM评估基准，测试模型从诊断报告生成可执行恢复脚本的能力。
MicroRemed: Benchmarking LLMs in Microservices Remediation
- 构建端到端微服务自愈评估基准，要求LLM直接生成Ansible剧本。
- 现有LLM在基准上表现不佳，需依赖多智能体迭代推理提升性能。
- 适合关注AI运维、自动化故障修复的研究者和工程师。
将大型语言模型（LLMs）与基于代理的推理框架结合，已在自主决策与系统级操作中展现出强大潜力。一个有前景但研究不足的方向是微服务自愈，即自动恢复故障微服务系统。现有方法仍依赖运维工程师手动编写提示词，仅让LLM将文本指令转换为可执行代码。为推动该领域发展，我们提出MicroRemed——首个用于评估LLM在端到端微服务自愈任务中表现的基准，要求模型从诊断报告直接生成可执行的Ansible playbook以恢复系统功能。我们进一步提出ThinkRemed，一种模拟运维工程师反思与感知能力的多智能体框架。实验表明，MicroRemed对当前LLM构成显著挑战，而ThinkRemed通过迭代推理与系统反思提升了整体修复性能。基准数据集已公开于https://github.com/LLM4AIOps/MicroRemed。
原文摘要 · Abstract (English)
Large Language Models (LLMs) integrated with agent-based reasoning frameworks have recently shown strong potential for autonomous decision-making and system-level operations. One promising yet underexplored direction is microservice remediation, where the goal is to automatically recover faulty microservice systems. Existing approaches, however, still rely on human-crafted prompts from Site Reliability Engineers (SREs), with LLMs merely converting textual instructions into executable code. To advance research in this area, we introduce MicroRemed, the first benchmark for evaluating LLMs in end-to-end microservice remediation, where models must directly generate executable Ansible playbooks from diagnosis reports to restore system functionality. We further propose ThinkRemed, a multi-agent framework that emulates the reflective and perceptive reasoning of SREs. Experimental results show that MicroRemed presents substantial challenges to current LLMs, while ThinkRemed improves end-to-end remediation performance through iterative reasoning and system reflection. The benchmark is available at https://github.com/LLM4AIOps/MicroRemed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。