arXiv:2602.21247cs.DBcs.DC2026-02KDD被引 1

PiPNN加速图结构近邻搜索索引构建,10分钟内完成十亿级数据索引。

PiPNN: Ultra-Scalable Graph-Based Nearest Neighbor Indexing

  • 采用动态哈希剪枝算法,分块并行计算距离,避免传统方法的搜索瓶颈。
  • 构建速度比Vamana快11.6倍、比HNSW快12.9倍,十亿级数据20分钟内完成。
  • 适合需要快速构建高精度近邻索引的大规模应用场景。

当前性能最优的近似最近邻搜索(ANN)索引多基于图结构,如HNSW和Vamana,但其构建速度慢,因依赖随机访问密集的束搜索。本文提出PiPNN(Pick-in-Partitions Nearest Neighbors),一种超可扩展的图构建算法,通过核心创新——HashPrune,动态维护稀疏边集合,实现数据集分块、批量距离计算(利用密集矩阵乘法核)与边流式输入。该机制保证构建过程内存有界,无需额外中间存储,可生成更高质量索引。PiPNN构建速度较Vamana(DiskANN)快11.6倍,较HNSW快12.9倍;相比MIRAGE快19.1倍,比FastKCNA快17.3倍,同时提升查询吞吐量。首次在单个多核机器上20分钟内完成十亿级数据的高质量ANN索引构建。

原文摘要 · Abstract (English)

The fastest indexes for Approximate Nearest Neighbor Search today are also the slowest to build: graph-based methods like HNSW and Vamana achieve state-of-the-art query performance but have large construction times due to relying on random-access-heavy beam searches. We introduce PiPNN (Pick-in-Partitions Nearest Neighbors), an ultra-scalable graph construction algorithm that avoids this ``search bottleneck'' that existing graph-based methods suffer from. PiPNN's core innovation is HashPrune, a novel online pruning algorithm which dynamically maintains sparse collections of edges. HashPrune enables PiPNN to partition the dataset into overlapping sub-problems, efficiently perform bulk distance comparisons via dense matrix multiplication kernels, and stream a subset of the edges into HashPrune. HashPrune guarantees bounded memory during index construction which permits PiPNN to build higher quality indices without the use of extra intermediate memory. PiPNN builds state-of-the-art indexes up to 11.6x faster than Vamana (DiskANN) and up to 12.9x faster than HNSW. PiPNN is significantly more scalable than recent algorithms for fast graph construction. PiPNN builds indexes at least 19.1x faster than MIRAGE and 17.3x than FastKCNA while producing indexes that achieve higher query throughput. PiPNN enables us to build, for the first time, high-quality ANN indexes on billion-scale datasets in under 20 minutes using a single multicore machine.

近邻搜索图索引高效构建大规模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。