arXiv:2505.12524cs.DBcs.LG2025-05被引 9

HAKES提升嵌入向量搜索吞吐量与召回率,支持高并发读写。

HAKES: Scalable Vector Database for Embedding Search Service

  • 两阶段索引设计:快速过滤+精炼,压缩向量降低开销
  • 动态查询终止机制,提升搜索效率;参数解耦支持写入不降性能
  • 分布式架构下吞吐量最高达基线16倍,适合大规模实时检索场景

现代深度学习模型将复杂数据转换为高维嵌入向量以捕捉语义。新兴应用如检索增强生成,依赖嵌入空间中的近似最近邻(ANN)搜索来查找相似数据。现有向量数据库使用索引实现高效ANN搜索,其中基于图的索引因低延迟和高召回率成为主流。但这类索引构建成本高,在并发读写负载下存在显著争用,且难以跨多服务器扩展。本文目标是构建在并发读写下兼具高吞吐与高召回的向量数据库。为此,提出一种具有显式两阶段设计的ANN索引:第一阶段使用高度压缩向量快速过滤,第二阶段精炼以保证召回率,并引入轻量级机器学习方法优化索引参数。设计早期终止检查,动态适应每个查询的搜索过程。通过解耦学习参数管理,实现在保持搜索性能的同时支持写入。最终设计出HAKES,一个在分离架构中部署新索引的分布式向量数据库。在由深度学习模型生成的高维嵌入数据集上,对12种先进索引和3个分布式向量数据库进行评估。结果表明,该索引在高召回区域及并发读写负载下优于基线,且系统可扩展,最高吞吐量达基线的16倍。HAKES项目已开源:https://www.comp.nus.edu.sg/~dbsystem/hakes/

原文摘要 · Abstract (English)

Modern deep learning models capture the semantics of complex data by transforming them into high-dimensional embedding vectors. Emerging applications, such as retrieval-augmented generation, use approximate nearest neighbor (ANN) search in the embedding vector space to find similar data. Existing vector databases provide indexes for efficient ANN searches, with graph-based indexes being the most popular due to their low latency and high recall in real-world high-dimensional datasets. However, these indexes are costly to build, suffer from significant contention under concurrent read-write workloads, and scale poorly to multiple servers. Our goal is to build a vector database that achieves high throughput and high recall under concurrent read-write workloads. To this end, we first propose an ANN index with an explicit two-stage design combining a fast filter stage with highly compressed vectors and a refine stage to ensure recall, and we devise a novel lightweight machine learning technique to fine-tune the index parameters. We introduce an early termination check to dynamically adapt the search process for each query. Next, we add support for writes while maintaining search performance by decoupling the management of the learned parameters. Finally, we design HAKES, a distributed vector database that serves the new index in a disaggregated architecture. We evaluate our index and system against 12 state-of-the-art indexes and three distributed vector databases, using high-dimensional embedding datasets generated by deep learning models. The experimental results show that our index outperforms index baselines in the high recall region and under concurrent read-write workloads. Furthermore, \namesys{} is scalable and achieves up to $16\times$ higher throughputs than the baselines. The HAKES project is open-sourced at https://www.comp.nus.edu.sg/~dbsystem/hakes/.

向量数据库嵌入搜索高并发分布式

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。