提出Yi系统,实现向量索引的高效高质就地更新
Efficient and Effective In-place Graph-based Vector Index Updates

- 采用任务分解与合并机制,支持向量索引就地更新
- 在8亿数据集上更新吞吐提升1.75倍,搜索并发吞吐提升1.8倍
- 内存占用仅需73%,适合实时向量数据系统部署
在大语言模型时代,高效向量更新对捕捉快速变化的数据至关重要。然而,频繁的向量插入与删除更新难以同时保证高吞吐与高召回率。传统基于聚类的索引方法虽有高更新吞吐但检索质量差;现有基于图的离线更新方法因需周期性合并更新批次,导致更新吞吐低。本文提出Yi系统,实现高效的就地图结构向量索引更新。核心思想为‘分解促进合并’,设计三组件:(i) 基于任务块的执行引擎,(ii) 异步缓冲管理器,(iii) 向量文件系统。实验表明,在8亿向量数据集上,Yi比当前最优系统更新吞吐高1.75倍,并发搜索吞吐高1.8倍,峰值内存使用仅73%,所需CPU核心更少。
原文摘要 · Abstract (English)
In the era of Large Language Models (LLMs), efficient vector updates are critical for capturing real-time information from rapidly evolving data. However, it is not trivial to process frequent vector insert and delete updates and maintain a high recall of the search results simultaneously. Specifically, the cluster-based vector indexing methods have high update throughput but low search result quality. Existing out-of-place graph-based vector indexing update approaches suffer from poor update throughput due to the need to periodically merge update batches into the underlying graph index. Building a vector data system that supports efficient and effective in-place updates is inherently challenging. In this work, we propose Yi to achieve it. In particular, Yi supports in-place graph-based vector indexing updates with consistently high update throughput and good search result quality. The key idea of Yi is decomposition facilitates consolidation. In particular, we introduce a vector-level update mechanism and architect Yi with three core components: (i) a tasklet-based execution engine, (ii) an asynchronous buffer manager, and (iii) a vector file system. Experimental results demonstrate that Yi achieves 1.75x higher update throughput and 1.8x higher concurrent search throughput than the state-of-the-art systems on the 800M dataset, while using only 73% of the peak memory and fewer CPU cores.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。