为现代嵌入检索设计的新基准，评估22种向量索引性能。
VIBE: Vector Index Benchmark for Embeddings
- 用现代嵌入模型生成真实场景数据集，支持检索增强生成等应用。
- 覆盖11个分布内和8个分布外数据集，涵盖多模态与近似注意力计算。
- 开源框架可复现，适合算法研究者与工程部署人员参考。
近似最近邻（ANN）搜索是众多机器学习流程中的关键组件，严格的基准测试对于评估向量索引性能至关重要。然而，现有基准的数据集已无法反映现代ANN应用场景，亟需更新。为此，我们提出面向嵌入的向量索引基准（VIBE），一个用于评测ANN算法的开源框架。VIBE提供了一套生成基准数据集的流程，采用代表现代应用的密集嵌入模型，包括检索增强生成（RAG）。为模拟真实工作负载，还引入了分布外（OOD）数据集，其中查询和语料库来自不同分布，涵盖多模态检索与最大内积搜索（MIPS）数据集，后者覆盖两个近期应用：近似注意力计算及多向量检索到单向量MIPS的降维。我们利用VIBE对22个开源向量索引实现进行了全面评估，覆盖11个分布内与8个分布外数据集。基准代码已开源，地址为 https://github.com/vector-index-bench/vibe。
原文摘要 · Abstract (English)
Approximate nearest neighbor (ANN) search is a performance-critical component of many machine learning pipelines, and rigorous benchmarking is essential for assessing the performance of vector indexes for ANN search. However, the datasets of existing benchmarks no longer represent modern ANN applications, creating a need for an up-to-date benchmark. To address this gap, we introduce Vector Index Benchmark for Embeddings (VIBE), an open-source framework for benchmarking ANN algorithms. VIBE provides a pipeline for generating benchmark datasets with dense embedding models representative of modern applications, including retrieval-augmented generation (RAG). To represent real-world workloads, we also include out-of-distribution (OOD) datasets where the queries and the corpus are drawn from different distributions. These include multimodal retrieval datasets and maximum inner product search (MIPS) datasets covering two recent use cases: approximate attention computation and reductions of multi-vector retrieval to single-vector MIPS. We use VIBE to conduct a comprehensive evaluation of 22 open-source vector-index implementations across 11 in-distribution and 8 out-of-distribution datasets. The benchmark is available at https://github.com/vector-index-bench/vibe
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。