arXiv:2504.18599cs.LG2025-04被引 1

用脑启发模型与统计检验结合,实时检测数据漂移和异常。

A Hybrid Framework for Real-Time Data Drift and Anomaly Identification Using Hierarchical Temporal Memory and Statistical Tests

  • 融合HTM与SPRT实现无训练周期的在线检测
  • 相比KS、Wasserstein等方法,准确率更高且计算更高效
  • 适合电信等需低误报率的实时监控场景

数据漂移指数据生成模型随时间变化,导致基于历史数据训练的模型逐渐失效。为应对这一问题,本文提出一种结合层级时间记忆(HTM)与序列概率比检验(SPRT)的混合框架,实现数据漂移与异常的实时识别。与传统方法不同,该框架无需频繁重训,且具备低误报率。HTM是一种类脑记忆模型,支持流式数据在线学习,无需独立训练测试阶段。本文还探索了多维监督场景下的应用:通过神经网络融合多个单维HTM列的输出,实现多维数据异常检测。实验表明,所提方法在准确性、适应性及计算效率上均优于传统的Kolmogorov-Smirnov检验、Wasserstein距离和群体稳定性指数(PSI)。研究还为电信等领域的真实部署提供了超参数优化建议。

原文摘要 · Abstract (English)

Data Drift is the phenomenon where the generating model behind the data changes over time. Due to data drift, any model built on the past training data becomes less relevant and inaccurate over time. Thus, detecting and controlling for data drift is critical in machine learning models. Hierarchical Temporal Memory (HTM) is a machine learning model developed by Jeff Hawkins, inspired by how the human brain processes information. It is a biologically inspired model of memory that is similar in structure to the neocortex, and whose performance is claimed to be comparable to state of the art models in detecting anomalies in time series data. Another unique benefit of HTMs is its independence from training and testing cycle; all the learning takes place online with streaming data and no separate training and testing cycle is required. In sequential learning paradigm, Sequential Probability Ratio Test (SPRT) offers some unique benefit for online learning and inference. This paper proposes a novel hybrid framework combining HTM and SPRT for real-time data drift detection and anomaly identification. Unlike existing data drift methods, our approach eliminates frequent retraining and ensures low false positive rates. HTMs currently work with one dimensional or univariate data. In a second study, we also propose an application of HTM in multidimensional supervised scenario for anomaly detection by combining the outputs of multiple HTM columns, one for each dimension of the data, through a neural network. Experimental evaluations demonstrate that the proposed method outperforms conventional drift detection techniques like the Kolmogorov-Smirnov (KS) test, Wasserstein distance, and Population Stability Index (PSI) in terms of accuracy, adaptability, and computational efficiency. Our experiments also provide insights into optimizing hyperparameters for real-time deployment in domains such as Telecom.

数据漂移异常检测脑启发模型实时系统

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。