arXiv:2501.01061stat.COcs.LG2025-01被引 5

提出高效在线异常检测算法,大幅降低计算开销且精度随数据量提升

An Efficient Outlier Detection Algorithm for Data Streaming

  • 仅重算新数据点的局部离群因子,不更新历史点分数
  • 在真实与模拟数据上,数据量越大越优于传统增量算法
  • 适合高吞吐实时场景,如金融风控、医疗监测

现代数据日益实时化,异常检测在金融反欺诈、医疗患者监护等领域至关重要。传统局部离群因子(LOF)算法因每次新增数据需重新计算全部得分,难以适应实时环境。虽有增量式LOF(ILOF)缓解此问题,但在大规模数据流下仍计算昂贵,且检测性能随数据量增长而下降。本文提出高效增量LOF(EILOF)算法,仅对新数据点重新计算LOF分数,不更新已有数据点的分数。尽管历史点分数未精确更新,但实际数据常含噪声,微小偏差未必影响检测效果,甚至可能提升性能。在模拟与真实数据集上的系统测试表明,随着数据流增大,EILOF在多种场景下均优于ILOF,不仅显著降低计算成本,还随新增数据量增加而系统性提升检测准确率。

原文摘要 · Abstract (English)

The nature of modern data is increasingly real-time, making outlier detection crucial in any data-related field, such as finance for fraud detection and healthcare for monitoring patient vitals. Traditional outlier detection methods, such as the Local Outlier Factor (LOF) algorithm, struggle with real-time data due to the need for extensive recalculations with each new data point, limiting their application in real-time environments. While the Incremental LOF (ILOF) algorithm has been developed to tackle the challenges of online anomaly detection, it remains computationally expensive when processing large streams of data points, and its detection performance may degrade after a certain threshold of points have streamed in. In this paper, we propose a novel approach to enhance the efficiency of LOF algorithms for online anomaly detection, named the Efficient Incremental LOF (EILOF) algorithm. The EILOF algorithm only computes the LOF scores of new points without altering the LOF scores of existing data points. Although exact LOF scores have not yet been computed for the existing points in the new algorithm, datasets often contain noise, and minor deviations in LOF score calculations do not necessarily degrade detection performance. In fact, such deviations can sometimes enhance outlier detection. We systematically tested this approach on both simulated and real-world datasets, demonstrating that EILOF outperforms ILOF as the volume of streaming data increases across various scenarios. The EILOF algorithm not only significantly reduces computational costs, but also systematically improves detection accuracy when the number of additional points increases compared to the ILOF algorithm.

异常检测数据流实时计算优化算法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。