提出新聚类评估方法，高维数据下更稳定可靠。
High-Dimensional BWDM: A Robust Nonparametric Clustering Validation Index for Large-Scale Data
- 结合随机投影与主成分分析，缓解高维数据问题。
- 在高维和含异常值场景下仍保持稳定评估性能。
- 适合现代高维非参数聚类任务，计算高效。
确定无监督学习中合适的聚类数量是统计学与数据科学的核心问题。传统有效性指标如Calinski-Harabasz、Silhouette和Davies-Bouldin依赖于中心点距离，在高维或含噪声数据中表现下降。本文提出一种新的鲁棒非参数聚类验证框架——高维间-内距离中位数（HD-BWDM），将近期提出的BWDM准则扩展至高维空间。该方法融合随机投影与主成分分析以缓解维度灾难，并采用截尾聚类与中位数距离增强对异常值的鲁棒性。理论分析表明，在Johnson-Lindenstrauss嵌入下具有一致性和收敛性。大量模拟实验显示，HD-BWDM在高维投影与数据污染下仍保持稳定且可解释，为传统基于中心点的验证方法提供了稳健替代方案。该方法为现代高维应用中的非参数聚类提供理论支持与高效停止规则。
原文摘要 · Abstract (English)
Determining the appropriate number of clusters in unsupervised learning is a central problem in statistics and data science. Traditional validity indices such as Calinski-Harabasz, Silhouette, and Davies-Bouldin-depend on centroid-based distances and therefore degrade in high-dimensional or contaminated data. This paper proposes a new robust, nonparametric clustering validation framework, the High-Dimensional Between-Within Distance Median (HD-BWDM), which extends the recently introduced BWDM criterion to high-dimensional spaces. HD-BWDM integrates random projection and principal component analysis to mitigate the curse of dimensionality and applies trimmed clustering and medoid-based distances to ensure robustness against outliers. We derive theoretical results showing consistency and convergence under Johnson-Lindenstrauss embeddings. Extensive simulations demonstrate that HD-BWDM remains stable and interpretable under high-dimensional projections and contamination, providing a robust alternative to traditional centroid-based validation criteria. The proposed method provides a theoretically grounded, computationally efficient stopping rule for nonparametric clustering in modern high-dimensional applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。