用规则增强的过滤方法自动清理日志,提升异常检测准确率。
LogPurge: Log Data Purification for Anomaly Detection via Rule-Enhanced Filtering
- 结合大模型与系统规则,分两阶段过滤日志中的异常模式。
- 在三个数据集上平均清除98.74%异常,保留82.39%正常日志。
- 适合需要高精度日志分析的运维与安全团队使用。
日志异常检测对识别系统故障和预防安全攻击至关重要,需从海量日志中发现异常模式,广泛应用于服务可靠性、性能优化和数据库日志分析。现有方法依赖干净无异常的日志序列训练深度学习模型,但获取此类数据需大量人工标注,且现有自动清洗方法未能充分融合日志的特定特征与实际语义。本文提出成本感知、规则增强的净化框架LogPurge,可从含噪日志中自动筛选出足够数量的正常日志序列用于训练。该方法采用两阶段过滤:第一阶段利用大语言模型(LLM)剔除聚集性异常模式,并通过增强系统规则提升LLM对系统日志的理解;第二阶段采用分治策略,将剩余污染区域分解为更小子问题,再通过第一阶段流程逐一净化。在两个公开数据集和一个工业数据集上的实验表明,该方法平均清除98.74%异常,同时保留82.39%正常样本。相比最新无监督样本选择算法,其在公开数据集上F-1分数分别提升35.7%和84.11%,在私有数据集上提升高达149.72%,充分验证了方法有效性。
原文摘要 · Abstract (English)
Log anomaly detection, which is critical for identifying system failures and preempting security breaches, detects irregular patterns within large volumes of log data, and impacts domains such as service reliability, performance optimization, and database log analysis. Modern log anomaly detection methods rely on training deep learning models on clean, anomaly-free log sequences. However, obtaining such clean log data requires costly and tedious human labeling, and existing automatic cleaning methods fail to fully integrate the specific characteristics and actual semantics of logs in their purification process. In this paper, we propose a cost-aware, rule-enhanced purification framework, LogPurge, that automatically selects a sufficient subset of normal log sequences from contamination log sequences to train a anomaly detection model. Our approach involves a two-stage filtering algorithm: In the first stage, we use a large language model (LLM) to remove clustered anomalous patterns and enhance system rules to improve LLM's understanding of system logs; in the second stage, we utilize a divide-and-conquer strategy that decomposes the remaining contaminated regions into smaller subproblems, allowing each to be effectively purified through the first stage procedure. Our experiments, conducted on two public datasets and one industrial dataset, show that our method significantly removes an average of 98.74% of anomalies while retaining 82.39% of normal samples. Compared to the latest unsupervised log sample selection algorithms, our method achieves F-1 score improvements of 35.7% and 84.11% on the public datasets, and an impressive 149.72% F-1 improvement on the private dataset, demonstrating the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。