arXiv:2504.02994cs.LGcs.DB2025-04被引 2

用强化学习动态调整日志异常检测阈值,提升准确率。

Improving log-based anomaly detection through learned adaptive filter

  • 用深度强化学习自动学习不同日志序列的过滤阈值
  • 在HDFS和BGL数据集上显著优于固定阈值方法
  • 适合需要高精度日志监控的系统运维人员

日志消息记录了系统运行的关键信息,对检测异常行为和管理现代软件系统具有重要意义。近年来,众多有监督和无监督学习方法被提出用于日志异常检测。当前最先进的无监督方法基于日志序列预测下一个日志事件,并采用固定配置(即使用相同的k值,将前k个预测事件视为正常)进行过滤,这导致检测性能下降,因为所有日志序列都使用同一个固定的k值,忽略了不同日志序列之间的动态特性和差异性。深度强化学习(DRL)在动态环境中做出智能决策方面得到广泛应用。本文认为,针对不同日志序列应采用自适应过滤机制。为此,我们提出一种基于DRL的新方法,构建可学习的自适应过滤器,为不同日志序列设定不同的正常/异常过滤阈值。我们将该问题建模为马尔可夫决策过程(MDP),并用DRL求解。我们在两个主流无监督日志异常检测方法DeepLog和LogAnomaly上,在HDFS和BGL两个数据集上进行了评估。大量实验表明,所提方法优于固定配置,显著提升了日志异常检测性能。

原文摘要 · Abstract (English)

Log messages record important system runtime information and are useful for detecting anomalous behaviors and managing modern software systems. Many supervised and unsupervised learning methods have been proposed recently for log-based anomaly detection. State-of-the-art unsupervised methods predict the next log event given a log sequence and apply fixed configurations that use the same filter condition (i.e. k, the top k predicted log events will be regarded as normal next events) which leads to inferior performance in the detection stage because it sets one fixed k for all log sequences, which ignores the dynamic nature and variance in different log sequences. Recently, deep reinforcement learning (DRL) are widely applied to make intelligent decisions in a dynamic environment. In this work, we contend that it is necessary to apply adaptive filters for different log sequences. To achieve this, we propose a novel approach based on DRL to construct a learned adaptive filter and apply different normal/abnormal filter thresholds for different log sequences. We define the Markov Decision Process (MDP) and formulate the learned adaptive filter as a problem that can be solved by DRL. We evaluate the learned adaptive filter on two state-of-the-art log-based anomaly detection unsupervised approaches DeepLog and LogAnomaly in two datasets HDFS and BGL. Extensive experiments show that our approach outperforms the fixed configurations and achieves significantly better performance in log-based anomaly detection.

日志分析异常检测强化学习自适应阈值

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。