arXiv:2411.12229cs.DBcs.IR2024-11中稿 · SIGMOD 2025被引 32

通过融合量化与图结构,提升高维近邻搜索效率与精度。

SymphonyQG: Towards Symphonious Integration of Quantization and Graph for Approximate Nearest Neighbor Search

  • 将量化码与图结构深度协同设计,实现批量距离估算
  • 避免额外重排序步骤,减少随机内存访问开销
  • 在真实数据集上达到当前最优的时间-精度平衡

高维欧氏空间中的近似最近邻(ANN)搜索应用广泛。现有基于图的方法在时间-精度权衡上表现优异,但受限于图索引搜索过程引发的随机内存访问以及精确距离计算带来的开销。为缓解这些问题,近期方法NGT-QG尝试融合量化与图结构:(1) 将顶点邻居的量化码紧凑复制存储,实现顺序访问;(2) 使用基于SIMD的FastScan批量估算距离以指导搜索。尽管如此,其仍未能充分发挥量化与图融合的潜力,例如需额外重排序步骤引入随机访问,且图结构未针对FastScan的批处理特性优化,导致搜索时计算浪费。本文提出SymphonyQG,进一步实现量化与图的更协调整合:无需显式重排序,并重构图结构以更好地适配FastScan的批处理机制。在多个真实数据集上的大量实验表明,SymphonyQG在时间-精度权衡上达到了新的最先进水平。

原文摘要 · Abstract (English)

Approximate nearest neighbor (ANN) search in high-dimensional Euclidean space has a broad range of applications. Among existing ANN algorithms, graph-based methods have shown superior performance in terms of the time-accuracy trade-off. However, they face performance bottlenecks due to the random memory accesses caused by the searching process on the graph indices and the costs of computing exact distances to guide the searching process. To relieve the bottlenecks, a recent method named NGT-QG makes an attempt by integrating quantization and graph. It (1) replicates and stores the quantization codes of a vertex's neighbors compactly so that they can be accessed sequentially, and (2) uses a SIMD-based implementation named FastScan to efficiently estimate distances based on the quantization codes in batch for guiding the searching process. While NGT-QG achieves promising improvements over the vanilla graph-based methods, it has not fully unleashed the potential of integrating quantization and graph. For instance, it entails a re-ranking step to compute exact distances at the end, which introduces extra random memory accesses; its graph structure is not jointly designed considering the in-batch nature of FastScan, which causes wastes of computation in searching. In this work, following NGT-QG, we present a new method named SymphonyQG, which achieves more symphonious integration of quantization and graph (e.g., it avoids the explicit re-ranking step and refines the graph structure to be more aligned with FastScan). Based on extensive experiments on real-world datasets, SymphonyQG establishes the new state-of-the-art in terms of the time-accuracy trade-off.

近邻搜索量化图结构高效检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。