通过清理日志事件，提升异常检测模型效率与准确率。
Reducing Events to Augment Log-based Anomaly Detection Models: An Empirical Study
- 提出LogCleaner，自动过滤冗余和反向日志事件。
- 日志量减少超70%，推理速度提升约300%。
- 适用于需要高效日志处理的系统监控场景。
随着软件系统日益复杂，精准检测异常变得至关重要但也充满挑战。现有基于日志的异常检测方法高度依赖大量日志数据，导致推理效率低下，并可能受噪声日志误导。然而，日志量缩减对检测效果的影响尚未被量化研究。为此，我们对六种不同模型在三个数据集上的表现进行了全面评估，揭示了三类不同类型的日志事件对模型性能的差异化影响。基于此，我们提出LogCleaner：一种用于异常检测中的日志事件自动精简方法。作为系统与模型之间的中间件，LogCleaner持续更新并过滤原始日志中的反向事件和重复事件。实验表明，LogCleaner可使日志量减少超过70%，将模型推理速度提升约300%，并普遍改善各类模型的异常检测性能。
原文摘要 · Abstract (English)
As software systems grow increasingly intricate, the precise detection of anomalies have become both essential and challenging. Current log-based anomaly detection methods depend heavily on vast amounts of log data leading to inefficient inference and potential misguidance by noise logs. However, the quantitative effects of log reduction on the effectiveness of anomaly detection remain unexplored. Therefore, we first conduct a comprehensive study on six distinct models spanning three datasets. Through the study, the impact of log quantity and their effectiveness in representing anomalies is qualifies, uncovering three distinctive log event types that differently influence model performance. Drawing from these insights, we propose LogCleaner: an efficient methodology for the automatic reduction of log events in the context of anomaly detection. Serving as middleware between software systems and models, LogCleaner continuously updates and filters anti-events and duplicative-events in the raw generated logs. Experimental outcomes highlight LogCleaner's capability to reduce over 70% of log events in anomaly detection, accelerating the model's inference speed by approximately 300%, and universally improving the performance of models for anomaly detection.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。