用预计算表快速去重，让近邻搜索结果又准又多样
LotusFilter: Fast Diverse Nearest Neighbor Search via a Learned Cutoff Table
- 预构建向量相似性表，搜索后快速剔除重复项
- 每查询仅需0.02毫秒，适合实时RAG场景
- 特别适合需要精准且多样的检索应用
近似最近邻搜索（ANNS）是RAG等应用的核心组件，但常产生过于相似的结果。在某些场景下，结果需既贴近查询又保持多样性。本文提出LotusFilter，一种后处理模块，通过预计算的截断表来识别并移除相近向量。搜索时，该模块贪婪查表剔除冗余候选。实验表明，其在模拟真实RAG环境下的运行速度极快，仅需0.02毫秒/查询，支持OpenAI嵌入等特征。代码已开源：https://github.com/matsui528/lotf。
原文摘要 · Abstract (English)
Approximate nearest neighbor search (ANNS) is an essential building block for applications like RAG but can sometimes yield results that are overly similar to each other. In certain scenarios, search results should be similar to the query and yet diverse. We propose LotusFilter, a post-processing module to diversify ANNS results. We precompute a cutoff table summarizing vectors that are close to each other. During the filtering, LotusFilter greedily looks up the table to delete redundant vectors from the candidates. We demonstrated that the LotusFilter operates fast (0.02 [ms/query]) in settings resembling real-world RAG applications, utilizing features such as OpenAI embeddings. Our code is publicly available at https://github.com/matsui528/lotf.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。