提出新方法评估大规模聚类解的优劣,误差低于3%。
Strong bounds for large-scale Minimum Sum-of-Squares Clustering
- 用分治法分解问题,结合反聚类启发式求解。
- 在大规模数据上实现小于3%的最优性差距。
- 适合需要验证聚类质量的研究者与工程师。
聚类是数据分析与机器学习中的基础技术,用于将相似数据点分组。最小平方和聚类(MSSC)是最常用的方法之一,旨在最小化数据点与其簇中心之间欧氏距离的平方和。由于聚类的无监督特性,达到全局最优至关重要但计算困难,求解复杂度随数据点数量呈指数增长,使精确方法难以应用于大规模数据集。即使获得最优目标值的强下界也计算成本高昂,难以评估启发式解的质量。本文提出一种新方法,通过最优性差距验证启发式MSSC解的有效性。该方法采用分治策略,将问题分解为可由精确求解器处理的小规模实例,分解过程由辅助优化问题——反聚类问题引导,为此设计了高效启发式算法。计算实验表明,该方法在大规模实例中表现优异,实现低于3%的最优性差距,且计算时间合理。结果证明了该方法在评估大规模聚类解质量方面的实用性,填补了MSSC评估中的关键空白。
原文摘要 · Abstract (English)
Clustering is a fundamental technique in data analysis and machine learning, used to group similar data points together. Among various clustering methods, the Minimum Sum-of-Squares Clustering (MSSC) is one of the most widely used. MSSC aims to minimize the total squared Euclidean distance between data points and their corresponding cluster centroids. Due to the unsupervised nature of clustering, achieving global optimality is crucial, yet computationally challenging. The complexity of finding the global solution increases exponentially with the number of data points, making exact methods impractical for large-scale datasets. Even obtaining strong lower bounds on the optimal MSSC objective value is computationally prohibitive, making it difficult to assess the quality of heuristic solutions. We address this challenge by introducing a novel method to validate heuristic MSSC solutions through optimality gaps. Our approach employs a divide-and-conquer strategy, decomposing the problem into smaller instances that can be handled by an exact solver. The decomposition is guided by an auxiliary optimization problem, the "anticlustering problem", for which we design an efficient heuristic. Computational experiments demonstrate the effectiveness of the method for large-scale instances, achieving optimality gaps below 3% while maintaining reasonable computational times. These results highlight the practicality of our approach in assessing feasible clustering solutions for large datasets, bridging a critical gap in MSSC evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。