arXiv:2607.01993cs.DScs.DC2026-07

提出首个可证明精度的聚类轮廓近似算法,支持大规模分布式计算。

Scalable and Distributed Silhouette Approximation

  • 基于采样设计,用子线性距离计算估算每个点和整体轮廓值。
  • 误差可控,保证在ε以内,概率不低于1-δ,参数可调。
  • 适合超大规模数据集,支持MapReduce/MPC框架高效分布式部署。

轮廓系数是评估数据集n个元素k-聚类质量的常用指标,其精确计算需Θ(n²)次距离计算,在现代大数据场景下效率极低。现有近似方法虽使用O(n²)距离计算,但仅为启发式且无理论精度保障。本文首次提出严格且高效的算法,通过采样实现对每个数据点局部轮廓及整体轮廓的估计,仅需O(nkε⁻²ln(nk/δ))次距离计算,即可在概率1-δ下保证加性误差不超过O(ε)。同时,设计了适用于MapReduce和大规模并行计算(MPC)框架的可扩展分布式版本,仅需常数轮次与亚线性本地内存。实验表明,本方法在局部与全局轮廓估计上均实现最佳准确率与效率平衡,可有效处理无法进行精确计算的大规模数据集。

原文摘要 · Abstract (English)

The silhouette is one of the most widely used measures to assess the quality of a $k$-clustering of a dataset of $n$ elements. Its evaluation requires no information beyond the clustering assignment. In addition, the silhouette is extremely easy to interpret, providing a score to measure the quality of a clustering as a whole or for each element. The exact computation of the: (i) silhouette of each element of a dataset; and (ii) the global silhouette of the clustering; require $Θ(n^2)$ distance calculations, under general metrics. The quadratic complexity $Θ(n^2)$ is extremely prohibitive, especially on massive modern datasets. Surprisingly, existing approximate methods using $O(n^2)$ distance calculations are heuristics not offering provable and controllable guarantees on the quality of their results. We introduce the first rigorous and efficient algorithms to estimate: (i) the (local) silhouette of each element of a dataset; and (ii) the (global) silhouette; of any metric $k$-clustering. Our methods, based on sampling, perform $O(nk\varepsilon^{-2}\ln (nk/δ))$ distance computations, and provide estimates with additive error $O(\varepsilon)$ with probability at least $1-δ$. That is, parameters $\varepsilon$ and $δ$ in $(0,1)$ control the trade-off between accuracy and efficiency. We also introduce a scalable and distributed design of our methods for the MapReduce and Massively Parallel Computing (MPC) frameworks. Our distributed algorithms use a constant number of rounds and sublinear local memory. Finally, we perform extensive experiments against state-of-the-art approaches. The results show that our new techniques yield the best trade-off between accuracy and efficiency for both local and global silhouette estimation. In addition, our methods scale efficiently to massive datasets for which an exact computation of the silhouette is not practical.

聚类评估轮廓系数分布式计算采样算法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。