arXiv:2505.12751cs.LGcs.AI2025-05

基于数据流形结构的异常检测与聚类方法,提升真实场景下异常识别准确率。

Structure-based Anomaly Detection and Clustering

  • 将数据映射到高维偏好空间,通过几何与哈希机制分离异常点
  • 在合成与真实数据集上优于现有方法,对噪声和初始采样不敏感
  • 适用于实时数据流与网络安全场景,可直接部署于生产系统

异常检测是医疗、制造和网络安全等领域的基础问题。本文提出针对结构化数据与流式数据的新无监督检测方法。第一部分聚焦基于流形结构的异常检测:正常数据服从低维流形,异常则偏离该结构。提出偏好隔离森林(PIF),通过流形拟合将数据嵌入高维偏好空间,并设计两种变体——基于几何距离的Voronoi-iForest与利用局部敏感哈希的RuzHash-iForest,以实现高效异常隔离。进一步提出Sliding-PIF,用于捕捉流式数据中的局部流形信息。实验表明,该方法在合成与真实数据集上均优于现有技术。扩展至结构化聚类,提出MultiLink方法,可从含噪数据中恢复多类几何模型,采用模型感知的链接策略合并簇,具有速度快、对阈值不敏感、抗初始采样差等优势。第二部分研究演化数据流中的在线异常检测,提出Online-iForest,使用自适应多分辨率直方图与动态更新树结构,无需重新训练即可跟踪变化,精度接近离线模型且效率更高。最后,针对网络安全中的恶意软件分类,改进梯度提升分类器,引入MaxLogit实现开放集识别,可检测未见恶意家族,该方法已集成至Cleafy生产系统。

原文摘要 · Abstract (English)

Anomaly detection is a fundamental problem in domains such as healthcare, manufacturing, and cybersecurity. This thesis proposes new unsupervised methods for anomaly detection in both structured and streaming data settings. In the first part, we focus on structure-based anomaly detection, where normal data follows low-dimensional manifolds while anomalies deviate from them. We introduce Preference Isolation Forest (PIF), which embeds data into a high-dimensional preference space via manifold fitting, and isolates outliers using two variants: Voronoi-iForest, based on geometric distances, and RuzHash-iForest, leveraging Locality Sensitive Hashing for scalability. We also propose Sliding-PIF, which captures local manifold information for streaming scenarios. Our methods outperform existing techniques on synthetic and real datasets. We extend this to structure-based clustering with MultiLink, a novel method for recovering multiple geometric model families in noisy data. MultiLink merges clusters via a model-aware linkage strategy, enabling robust multi-class structure recovery. It offers key advantages over existing approaches, such as speed, reduced sensitivity to thresholds, and improved robustness to poor initial sampling. The second part of the thesis addresses online anomaly detection in evolving data streams. We propose Online Isolation Forest (Online-iForest), which uses adaptive, multi-resolution histograms and dynamically updates tree structures to track changes over time. It avoids retraining while achieving accuracy comparable to offline models, with superior efficiency for real-time applications. Finally, we tackle anomaly detection in cybersecurity via open-set recognition for malware classification. We enhance a Gradient Boosting classifier with MaxLogit to detect unseen malware families, a method now integrated into Cleafy's production system.

异常检测流形学习在线学习网络安全

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。