为图数据库设计高效向量索引,支持任意筛选条件下的近邻搜索。
NaviX: A Native Vector Index Design for Graph DBMSs With Robust Predicate-Agnostic Search Performance
- 基于HNSW图构建原生向量索引,复用图数据库存储与查询能力。
- 提出自适应预过滤算法,在不同数据相关性下保持稳定性能。
- 适合需要联合查询结构数据与向量嵌入的场景,如推荐系统。
为满足现代预测应用对统一数据系统的日益增长需求,需扩展传统数据库管理系统(DBMS)以支持向量索引,实现对对象的结构属性与连接关系及向量嵌入的联合查询。本文提出NaviX,一种专为图数据库管理系统(GDBMS)设计的原生向量索引,具有两大目标:首先,构建基于磁盘的向量索引,充分利用底层GDBMS的核心存储与查询处理能力,采用层次可导航小世界(HNSW)图作为基础结构;其次,支持谓词无关的过滤向量搜索,即在由任意选择子查询QS定义的向量子集S中查找查询向量vQ的k个最近邻(kNN)。为此,我们采用预过滤策略,先评估QS,再将子集S的完整描述传递给kNN搜索算子。研究了如何设计一种在不同选择率和子集S与查询向量vQ间相关性变化下仍具鲁棒性的预过滤算法。提出一种自适应算法,利用HNSW图中每个向量的局部选择率,在kNN搜索每一步动态选择合适的启发式策略。通过大量实验,证明了NaviX在效率和鲁棒性上优于现有基于预过滤与后过滤的基准方法。
原文摘要 · Abstract (English)
There is an increasing demand for extending existing DBMSs with vector indices so that they become unified systems capable of supporting modern predictive applications, which require joint querying of vector embeddings together with the structured properties and connections of objects. We present NaviX, a native vector index for graph DBMSs (GDBMSs) that has two main design goals. First, we aim to implement a disk-based vector index that leverages the core storage and query-processing capabilities of the underlying GDBMS. To this end, NaviX is built on the Hierarchical Navigable Small-World (HNSW) graph, which itself is a graph-based structure. Second, we aim to support predicate-agnostic filtered vector search queries, in which the k nearest neighbors (kNNs) of a query vector vQ are searched only within an arbitrary subset S of vectors defined by an ad-hoc selection sub-query QS. We adopt a prefiltering approach that evaluates QS first and passes the full description of subset S to the kNN search operator. We study how to design a prefiltering search algorithm that remains robust under varying selectivities and under different correlations between subset S and query vector vQ. We propose an adaptive algorithm that uses the local selectivity of each vector in the HNSW graph to choose an appropriate heuristic at every iteration of the kNN search. Finally, We demonstrate NaviX's robustness and efficiency through extensive experiments against both existing prefiltering- and postfiltering-based baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。