百亿级数据下用CPU实现快速带多维过滤的相似搜索
Billion-scale Similarity Search Using a Hybrid Indexing Approach with Advanced Filtering
- 扩展IVF-Flat结构，融合稠密向量与离散过滤属性
- 在百亿规模数据上实现低延迟检索，支持复杂过滤条件
- 专为CPU设计，成本低，适合实际部署场景
本文提出一种新型相似搜索方法，可在百亿规模数据集上实现带复杂过滤功能的高效检索，专为CPU推理优化。该方法扩展经典IVF-Flat索引结构，整合高维稠密嵌入与离散过滤属性，实现高维空间中的快速检索。所提方案基于磁盘存储，针对CPU系统设计，提供一种经济高效的大型相似搜索解决方案。通过案例研究验证了其有效性，展示了在多种实际应用场景中的潜力。
原文摘要 · Abstract (English)
This paper presents a novel approach for similarity search with complex filtering capabilities on billion-scale datasets, optimized for CPU inference. Our method extends the classical IVF-Flat index structure to integrate multi-dimensional filters. The proposed algorithm combines dense embeddings with discrete filtering attributes, enabling fast retrieval in high-dimensional spaces. Designed specifically for CPU-based systems, our disk-based approach offers a cost-effective solution for large-scale similarity search. We demonstrate the effectiveness of our method through a case study, showcasing its potential for various practical uses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。