改进图索引的范围检索,速度最快提升100倍。
Range Retrieval with Graph-Based Indices
- 基于图搜索改造算法,动态适应查询结果数量
- 在百亿数据集上实现最高100倍查询吞吐提升
- 适合大规模相似性搜索场景,如重复检测
高维向量空间中的邻近点检索是信息检索的关键步骤。尽管近似最近邻搜索(ANNS)已得到广泛研究,但与其相关的范围检索问题——即找出与查询点距离在指定阈值内的所有点——仍较少被关注,尽管其在去重、抄袭检测和人脸识别等领域有重要应用。本文提出针对图索引的新范围检索技术,针对范围查询结果从零到数千不等的特点,设计自适应算法:对无结果或少量结果的查询快速终止,对大量结果的查询投入更多资源。由于缺乏现有基准,我们系统分析了八个主流嵌入数据集的范围特性,并为每个数据集选定合适的检索半径,涵盖最多达十亿个数据点。实验表明,新算法在这些数据集上相比标准图搜索和FAISS-IVF范围搜索,查询吞吐最高提升100倍;相比先前改进的束搜索方法,性能提升达10倍,且在百亿级数据下仍表现优异。
原文摘要 · Abstract (English)
Retrieving points based on proximity in a high-dimensional vector space is a crucial step in information retrieval applications. The approximate nearest neighbor search (ANNS) problem, which identifies the $k$ nearest neighbors for a query, has been extensively studied in recent years. However, comparatively little attention has been paid to the related problem of finding all points within a given distance of a query, the range retrieval problem, despite its applications in areas such as duplicate detection, plagiarism checking, and facial recognition. In this paper, we present new techniques for range retrieval on graph-based vector indices, which are known to achieve excellent performance on ANNS queries. Since a range query may have anywhere from no matching results to thousands of matching results in the database, we introduce a set of range retrieval algorithms based on modifications of the standard graph search that adapt to terminate quickly on queries in the former group, and to put more resources into finding results for the latter group. Due to the lack of existing benchmarks for range retrieval, we also undertake a comprehensive study of range characteristics of existing embedding datasets, and select a suitable range retrieval radius for eight existing datasets with up to 1 billion points in addition to one existing benchmark. We test our algorithms on these datasets, and find up to 100x improvement in query throughput over a standard graph search and the FAISS-IVF range search algorithm. We also find up to 10x improvement over a previously suggested modification of the standard beam search, and strong performance up to 1 billion data points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。