捕捉流数据周期模式与异常,实时分析且可解释。
SDOoop: Capturing Periodical Patterns and Out-of-phase Anomalies in Streaming Data Analysis
- 通过保留数据结构时序信息,识别传统方法难发现的上下文异常。
- 在真实网络数据中揭示关键基础设施动态,性能优于或等同于顶尖方法。
- 适合需实时、可解释分析的物联网、网络安全等场景。
流数据处理在物联网、网络安全、机器人及工业系统中日益重要,但仍面临诸多挑战。尽管如此,其研究仍属新兴领域。现有方法SDO具备高速、可解释和直观参数化的特点,而本文提出SDOoop,扩展了其流版本对数据结构时序信息的保持能力。SDOoop能检测传统算法无法识别的上下文异常,同时支持数据几何、聚类与时间模式的可视化分析。我们在关键基础设施的真实网络通信数据上应用SDOoop,成功提取出反映系统动态的规律模式。此外,在入侵检测与自然科学研究领域的多组数据上评估,结果表明其性能等同或优于当前最优方法。实验验证了基于模型的新方法在流数据建模与解释方面的巨大潜力。由于SDOoop具有恒定的单样本空间与时间复杂度,适用于大数据场景,可即时处理海量信息。该方法符合下一代机器学习的要求,不仅追求高准确率与速度,更强调模型的可解释性与信息丰富性。
原文摘要 · Abstract (English)
Streaming data analysis is increasingly required in applications, e.g., IoT, cybersecurity, robotics, mechatronics or cyber-physical systems. Despite its relevance, it is still an emerging field with open challenges. SDO is a recent anomaly detection method designed to meet requirements of speed, interpretability and intuitive parameterization. In this work, we present SDOoop, which extends the capabilities of SDO's streaming version to retain temporal information of data structures. SDOoop spots contextual anomalies undetectable by traditional algorithms, while enabling the inspection of data geometries, clusters and temporal patterns. We used SDOoop to model real network communications in critical infrastructures and extract patterns that disclose their dynamics. Moreover, we evaluated SDOoop with data from intrusion detection and natural science domains and obtained performances equivalent or superior to state-of-the-art approaches. Our results show the high potential of new model-based methods to analyze and explain streaming data. Since SDOoop operates with constant per-sample space and time complexity, it is ideal for big data, being able to instantly process large volumes of information. SDOoop conforms to next-generation machine learning, which, in addition to accuracy and speed, is expected to provide highly interpretable and informative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。