arXiv:2608.05441cs.DBcs.DC2026-08

将向量搜索与文件剪枝结合,让过滤后向量查询快32倍。

Filtered Vector Search in a Disaggregated Lakehouse: Composing Table-Format Pruning with Per-File ANN

  • 在Parquet文件尾部嵌入IVF索引,用已有文件剪枝加速向量搜索。
  • 1150万条数据上,召回率90%时查询速度提升32倍。
  • 适合有文件级筛选条件的向量查询场景,尤其在湖仓架构中。

近似最近邻(ANN)搜索越来越多地与结构化数据结合,例如‘查找租户为acme且语言为en的10个最近文档’。然而,相似性搜索与过滤通常分别由专用向量索引和独立过滤步骤处理。本文探讨将两者集成到开放湖仓表(Apache Iceberg 上的 Parquet 文件)中的可能性。在每个 Parquet 文件尾部嵌入 IVF 索引,并利用现有文件剪枝机制(分区剪枝、区域映射、位图索引),先通过谓词剪枝数据文件,再仅在剩余文件上运行 IVF 搜索。该索引以元数据方式分布式构建,不破坏原有读取兼容性,配合基于接吻哈希的每文件缓存降低对象存储延迟。实验显示,在 1150 万 × 768 的数据集上,暖启动下召回率@10 ≥ 0.90 时,查询速度比暴力搜索快约 32 倍;在 500 万真实 IBM Granite 嵌入数据上,跨连接的过滤条件可剪除 4/5 区域分区,使查询时间从 14.7 秒降至 157 毫秒(约 94 倍加速)。研究明确了组合生效的前提:过滤列需具备文件级局部性,且谓词只能安全下推至经证明纯正(分区)的列,而非仅排序的列,并总结了直接叠加 ANN 到湖仓引擎时遇到的失败模式。

原文摘要 · Abstract (English)

Approximate nearest-neighbor (ANN) search increasingly runs alongside structured data - "find the 10 nearest documents where tenant='acme' AND lang='en'" - yet similarity and filtering are usually bolted together: a specialized vector index for one, a separate filter step for the other. We ask what happens when both live inside an open lakehouse table (Apache Iceberg over Parquet on object storage), where the engine already owns a mature file-pruning stack (partition pruning, zone-maps, a bitmap index). We embed an IVF index in place in each Parquet file's footer and make filtered vector queries fast not with a new filtering algorithm but by composing the table's existing file pruning with per-file ANN: the planner prunes data files by the predicate first, then runs IVF only over the survivors. The index is built distributed and non-destructively - a metadata-only Iceberg replace that every other engine still reads - and a rendezvous-hashed per-file cache keeps object-store read latency from swamping the algorithmic win. The payoff comes entirely from file pruning. On an 11.5M x 768 table, warm IVF search is ~32x faster than brute force at recall@10 >= 0.90, a selective predicate having pruned 355 of 444 data files before ANN runs; on 5M real IBM Granite embeddings, a filter arriving across a join prunes four of five region partitions and runs nearly two orders of magnitude (~94x: 14.7 s -> 157 ms) faster than the query-time join at identical top-k, once the reduction is materialized into a region-partitioned layout. We characterize when the composition pays off - it requires file-level locality on the filter column, and the residual predicate is only safe to push into the search over a provably pure (partitioned) column, not a merely sorted one - and report the failure modes we hit bolting ANN onto a lakehouse engine.

向量搜索湖仓一体文件剪枝IVF

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。