快速生成高度差异的分组,适用于大规模数据集的均衡划分。
A Fast and Effective Method for Euclidean Anticlustering: The Assignment-Based-Anticlustering Algorithm
- 基于分配策略构建新算法,高效处理百万级对象与数十万分组。
- 在数秒至数分钟内完成大规模任务,性能远超现有方法。
- 适合机器学习中的小批量生成、交叉验证等需要均衡分组的场景。
抗聚类(Anticlustering)是将一组对象划分为大小相等的反聚类(anticlusters),使同一反聚类内的对象尽可能不相似,以代表整体数据。本文研究欧氏距离下的平方欧氏距离作为相似性度量的场景。该问题在社会学研究、交叉验证、随机梯度下降的小批量生成以及平衡K-cut划分中有广泛应用。尤其在机器学习中,涉及百万级数据和极高数量的聚类数K,因此可扩展的抗聚类算法至关重要。本文提出一种新的分配式抗聚类(Assignment-Based Anticlustering, ABA)算法,可在数秒至数分钟内处理包含百万级对象和数十万反聚类的大规模实例,远超现有方法能力。通过大量计算实验表明,该算法在解的质量和运行时间上均优于现有方法,包括带类别约束的情况。对于平衡K-cut划分问题,其表现也优于著名的METIS方法。代码已开源于GitHub。
原文摘要 · Abstract (English)
Anticlustering is an NP-hard combinatorial optimization problem that consists of partitioning a set of objects into equal-sized groups called anticlusters such that the objects in the same anticluster are as dissimilar as possible and thereby representative of the entire set of objects. Here we study the case where the dissimilarity metric is the squared Euclidean distance between the respective feature vectors. Applications of Euclidean anticlustering include social studies, cross-validation, creating mini-batches for stochastic gradient descent, and finding balanced K-cut partitions. In particular, machine-learning applications such as mini-batch generation involve million-scale datasets and very large values of K, making scalable anticlustering algorithms essential. We propose a new algorithm, the Assignment-Based Anticlustering (ABA) algorithm, that scales to instances with millions of objects and hundreds of thousands of anticlusters within seconds to minutes, which is far beyond what existing anticlustering methods can manage. We demonstrate here, via an extensive computational study, that our algorithm outperforms existing anticlustering methods in both solution quality and running time. This is so also for anticlustering with categories. For the related problem of balanced K-cut partitioning, our algorithm is superior to the well-known METIS method. The code of our algorithm is available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。