用大模型自动标注日志，实现无需人工参与的异常检测
ALPHA: LLM-Enabled Active Learning for Human-Free Network Anomaly Detection
- 结合语义嵌入与聚类，用大模型少样本标注并传播标签
- 在真实数据集上达到与全监督方法相当的准确率
- 适合需要低成本自动化日志分析的运维与安全场景
网络日志分析在检测安全威胁和运行异常中至关重要。传统方法依赖专家知识或全监督学习模型，均需大量标注数据和人力投入。为此，我们提出ALPHA，首个无需人工参与的日志分析主动学习流程。ALPHA融合语义嵌入、基于聚类的代表性采样以及大语言模型（LLM）辅助的少样本标注，实现异常检测全流程自动化。通过将LLM生成的标签在聚类间传播，仅用极少标注即可训练大规模异常检测器。为提升标注精度，我们设计两步少样本精炼策略，根据LLM的错误模式自适应选择提示词。在多个真实世界日志数据集上的实验表明，ALPHA在检测准确率上媲美全监督方法，同时大幅降低人工参与。此外，其后检测阶段还能通过LLM生成可解释的根因分析报告。这些能力使ALPHA成为真正自动化的日志异常检测的可扩展、低成本解决方案。
原文摘要 · Abstract (English)
Network log data analysis plays a critical role in detecting security threats and operational anomalies. Traditional log analysis methods for anomaly detection and root cause analysis rely heavily on expert knowledge or fully supervised learning models, both of which require extensive labeled data and significant human effort. To address these challenges, we propose ALPHA, the first Active Learning Pipeline for Human-free log Analysis. ALPHA integrates semantic embedding, clustering-based representative sampling, and large language model (LLM)-assisted few-shot annotation to automate the anomaly detection process. The LLM annotated labels are propagated across clusters, enabling large-scale training of an anomaly detector with minimal supervision. To enhance the annotation accuracy, we propose a two-step few-shot refinement strategy that adaptively selects informative prompts based on the LLM's observed error patterns. Extensive experiments on real-world log datasets demonstrate that ALPHA achieves detection accuracy comparable to fully supervised methods while mitigating human efforts in the loop. ALPHA also supports interpretable analysis through LLM-driven root cause explanations in the post-detection stage. These capabilities make ALPHA a scalable and cost-efficient solution for truly automated log-based anomaly detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。