让多个专家模型协作解决AIOps问题,效果比单模型好得多。
CoE-Ops: Collaboration of LLM-based Experts for AIOps Question-Answering
- 用大模型分类任务,让不同专家分工处理高低层AIOps问题。
- 高阶任务路由准确率提升72%,问题解决准确率最高增8%。
- 适合需要多任务协同的运维智能系统研发人员参考。
随着人工智能快速发展,AIOps已成为DevOps领域的重要范式。现有工作虽提升了各阶段性能,但受限于领域知识,单一模型仅能处理特定任务(如日志解析、根因分析)。结合多个模型可实现更高效结果,这在集成学习与大模型训练中已得到验证。本文首次提出协作专家框架CoE-Ops,引入通用大语言模型任务分类器,并采用检索增强生成机制,提升对高阶(代码、构建、测试等)与低阶(故障分析、异常检测等)AIOps任务的处理能力。在DevOps-EVAL数据集上进行大量实验,结果表明:CoE-Ops相较现有CoE方法,在高阶任务路由准确率上提升72%;在问题解决准确率上较单模型最高提升8%;在准确率上优于更大规模的Mixture-of-Experts(MoE)模型达14%。
原文摘要 · Abstract (English)
With the rapid evolution of artificial intelligence, AIOps has emerged as a prominent paradigm in DevOps. Lots of work has been proposed to improve the performance of different AIOps phases. However, constrained by domain-specific knowledge, a single model can only handle the operation requirement of a specific task,such as log parser,root cause analysis. Meanwhile, combining multiple models can achieve more efficient results, which have been proved in both previous ensemble learning and the recent LLM training domain. Inspired by these works,to address the similar challenges in AIOPS, this paper first proposes a collaboration-of-expert framework(CoE-Ops) incorporating a general-purpose large language model task classifier. A retrieval-augmented generation mechanism is introduced to improve the framework's capability in handling both Question-Answering tasks with high-level(Code,build,Test,etc.) and low-level(fault analysis,anomaly detection,etc.). Finally, the proposed method is implemented in the AIOps domain, and extensive experiments are conducted on the DevOps-EVAL dataset. Experimental results demonstrate that CoE-Ops achieves a 72% improvement in routing accuracy for high-level AIOps tasks compared to existing CoE methods, delivers up to 8% accuracy enhancement over single AIOps models in DevOps problem resolution, and outperforms larger-scale Mixture-of-Experts (MoE) models by up to 14% in accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。