arXiv:2412.02244cs.LG2024-12被引 2

Dask-means高效压缩大规模空间向量,内存占用低于30MB,速度超传统算法168倍。

On Simplifying Large-Scale Spatial Vectors: Fast, Memory-Efficient, and Cost-Predictable k-means

  • 设计内存可调索引批量处理,加速近邻搜索
  • 在百万级数据上仅需<30MB内存,提速超168倍
  • 支持移动端部署,成本预测误差<3%

k-means算法可用于简化大规模空间向量(如2D地理坐标、3D点云),以支持快速分析与学习。然而,现有k-means算法在处理大规模数据时需大量计算资源,难以在资源受限设备上运行。本文提出一种快速、内存高效且成本可预测的k-means算法——Dask-means。首先,通过设计内存可调的加速器,利用优化的近邻搜索索引批量分配空间向量至聚类;其次,构建轻量级成本估算器,预测内存消耗与运行时间,使系统能根据约束请求合适内存或调整空间需求,并保证足够CPU时间。实验表明,在处理规模达$10^6$的数据集时,Dask-means内存占用低于30MB,较广泛使用的Lloyd算法提速超168倍。在移动设备上验证显示,其相较其他先进k-means算法具有显著速度优势与低内存开销。成本估算器预测内存误差小于3%,运行时间预测均方误差比现有方法降低最高达33.3%。

原文摘要 · Abstract (English)

The k-means algorithm can simplify large-scale spatial vectors, such as 2D geo-locations and 3D point clouds, to support fast analytics and learning. However, when processing large-scale datasets, existing k-means algorithms have been developed to achieve high performance with significant computational resources, such as memory and CPU usage time. These algorithms, though effective, are not well-suited for resource-constrained devices. In this paper, we propose a fast, memory-efficient, and cost-predictable k-means called Dask-means. We first accelerate k-means by designing a memory-efficient accelerator, which utilizes an optimized nearest neighbor search over a memory-tunable index to assign spatial vectors to clusters in batches. We then design a lightweight cost estimator to predict the memory cost and runtime of the k-means task, allowing it to request appropriate memory from devices or adjust the accelerator's required space to meet memory constraints, and ensure sufficient CPU time for running k-means. Experiments show that when simplifying datasets with scale such as $10^6$, Dask-means uses less than $30$MB of memory, achieves over $168$ times speedup compared to the widely-used Lloyd's algorithm. We also validate Dask-means on mobile devices, where it demonstrates significant speedup and low memory cost compared to other state-of-the-art (SOTA) k-means algorithms. Our cost estimator estimates the memory cost with a difference of less than $3\%$ from the actual ones and predicts runtime with an MSE up to $33.3\%$ lower than SOTA methods.

聚类算法内存优化成本预测空间向量

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。