无需参数设定,自动识别最佳聚类参数与异常值阈值。
Unsupervised Parameter-free Outlier Detection using HDBSCAN* Outlier Profiles
- 基于GLOSH得分范围动态选择最优minpts参数。
- 无需预设异常点数量,自动确定分类阈值。
- 完全无监督,适合缺乏先验知识的数据集。
在机器学习与数据挖掘中,异常值是显著偏离数据集的点,常引入无关信息并导致统计偏差与模型偏倚。因此,当缺乏标签信息时,无监督异常检测方法至关重要。现有方法GLOSH基于HDBSCAN*的层次结构,通过比较数据点密度与其所在区域最高密度来估算异常分数,但其性能受HDBSCAN*的minpts参数影响。由于缺乏对数据分布的先验知识,难以预先确定合适的minpts值,且传统方法需预先设定异常点数量n,这往往不切实际且未知。本文提出一种无监督策略,通过分析不同minpts取值下GLOSH得分的分布范围,自动选择能最好区分异常点与正常点的minpts值;同时提出无需预设任何值的阈值估计方法,实现完全自动化。实验表明,该方法可自动获得最优或近似最优的异常检测效果。
原文摘要 · Abstract (English)
In machine learning and data mining, outliers are data points that significantly differ from the dataset and often introduce irrelevant information that can induce bias in its statistics and models. Therefore, unsupervised methods are crucial to detect outliers if there is limited or no information about them. Global-Local Outlier Scores based on Hierarchies (GLOSH) is an unsupervised outlier detection method within HDBSCAN*, a state-of-the-art hierarchical clustering method. GLOSH estimates outlier scores for each data point by comparing its density to the highest density of the region they reside in the HDBSCAN* hierarchy. GLOSH may be sensitive to HDBSCAN*'s minpts parameter that influences density estimation. With limited knowledge about the data, choosing an appropriate minpts value beforehand is challenging as one or some minpts values may better represent the underlying cluster structure than others. Additionally, in the process of searching for ``potential outliers'', one has to define the number of outliers n a dataset has, which may be impractical and is often unknown. In this paper, we propose an unsupervised strategy to find the ``best'' minpts value, leveraging the range of GLOSH scores across minpts values to identify the value for which GLOSH scores can best identify outliers from the rest of the dataset. Moreover, we propose an unsupervised strategy to estimate a threshold for classifying points into inliers and (potential) outliers without the need to pre-define any value. Our experiments show that our strategies can automatically find the minpts value and threshold that yield the best or near best outlier detection results using GLOSH.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。