arXiv:2502.13826cs.IR2025-02被引 13

提出首个无需批量合并的图索引更新算法,支持高并发流式增删

In-Place Updates of a Graph Index for Streaming Approximate Nearest Neighbor Search

  • 通过原地更新机制处理每条插入/删除操作,避免批量合并
  • 在多种更新模式下保持稳定召回率,高低召回场景均有效
  • 相比现有方法提升查询吞吐与更新速度,适合实时检索系统

近似最近邻搜索(ANNS)索引是信息检索的核心组件,广泛应用于数据库、搜索、推荐和RAG系统。在这些场景中,文档或其他对象以高速率插入和删除,要求对向量索引进行流式更新。基于邻近图的索引算法在ANNS中最为高效,多次赢得基准测试冠军。然而,高频更新下保持稳定的召回率极具挑战性,尤其因图结构为单向链接,删除节点时难以快速定位其入邻居。因此,现有先进算法如FreshDiskANN采用累积删除并周期性批量合并的方式,移除指向已删除节点的边,并调整图结构以维持召回稳定性。本文提出IP-DiskANN(InPlaceUpdate-DiskANN),首个无需批量合并的算法,可高效原地处理每次插入与删除。实验表明,该算法在标准基准测试中,面对多种长时间更新模式,无论高召回或低召回场景,均保持稳定的召回率。此外,其查询吞吐与更新速度优于使用批量合并的算法及HNSW。

原文摘要 · Abstract (English)

Indices for approximate nearest neighbor search (ANNS) are a basic component for information retrieval and widely used in database, search, recommendation and RAG systems. In these scenarios, documents or other objects are inserted into and deleted from the working set at a high rate, requiring a stream of updates to the vector index. Algorithms based on proximity graph indices are the most efficient indices for ANNS, winning many benchmark competitions. However, it is challenging to update such graph index at a high rate, while supporting stable recall after many updates. Since the graph is singly-linked, deletions are hard because there is no fast way to find in-neighbors of a deleted vertex. Therefore, to update the graph, state-of-the-art algorithms such as FreshDiskANN accumulate deletions in a batch and periodically consolidate, removing edges to deleted vertices and modifying the graph to ensure recall stability. In this paper, we present IP-DiskANN (InPlaceUpdate-DiskANN), the first algorithm to avoid batch consolidation by efficiently processing each insertion and deletion in-place. Our experiments using standard benchmarks show that IP-DiskANN has stable recall over various lengthy update patterns in both high-recall and low-recall regimes. Further, its query throughput and update speed are better than using the batch consolidation algorithm and HNSW.

近似搜索图索引流式更新实时检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。