从任务视角揭示向量检索的三大隐性陷阱,指导实际应用选型。
Reveal Hidden Pitfalls and Navigate Next Generation of Vector Similarity Search from Task-Centric Views
- 提出任务中心评估框架,识别嵌入损失、度量误用、数据分布敏感三类性能衰减源。
- 在8个真实数据集上测试13种顶尖方法,发现传统排名与任务表现存在显著偏差。
- 提供可解释决策树和元特征,帮助开发者根据具体场景选择最优检索方案。
高维空间中的向量相似性搜索(VSS)正成为下一代数据库系统的核心功能,广泛应用于大模型嵌入查询、语义检索和推荐系统等场景。现有基准主要基于距离度量的召回率-延迟权衡评估,忽略了检索质量对下游任务的实际影响,导致学术与工业研究方向偏差。本文提出Iceberg,一个面向真实应用场景的端到端评估基准套件。从任务中心视角,揭示了信息丢失漏斗:(1) 特征提取阶段的嵌入损失;(2) 度量误用,即距离不能反映任务相关性;(3) 数据分布敏感性,体现索引在分布偏移与多模态下的鲁棒性。Iceberg覆盖图像分类、人脸识别、文本检索和推荐系统等8个领域,数据规模100万至1亿向量不等,包含丰富的任务标签与评估指标,支持在完整应用链路中评估检索算法。评测13种先进VSS方法并基于应用级指标重新排序,发现其与传统召回-延迟排名存在显著差异。基于此,我们定义一组任务中心元特征,并构建可解释决策树,指导从业者为特定工作负载选择与调优VSS方法。
原文摘要 · Abstract (English)
Vector Similarity Search (VSS) in high-dimensional spaces is rapidly emerging as core functionality in next-generation database systems for numerous data-intensive services -- from embedding lookups in large language models (LLMs), to semantic information retrieval and recommendation engines. Current benchmarks, however, evaluate VSS primarily on the recall-latency trade-off against a ground truth defined solely by distance metrics, neglecting how retrieval quality ultimately impacts downstream tasks. This disconnect can mislead both academic research and industrial practice. We present Iceberg, a holistic benchmark suite for end-to-end evaluation of VSS methods in realistic application contexts. From a task-centric view, Iceberg uncovers the Information Loss Funnel, which identifies three principal sources of end-to-end performance degradation: (1) Embedding Loss during feature extraction; (2) Metric Misuse, where distances poorly reflect task relevance; (3) Data Distribution Sensitivity, highlighting index robustness across skews and modalities. For a more comprehensive assessment, Iceberg spans eight diverse datasets across key domains such as image classification, face recognition, text retrieval, and recommendation systems. Each dataset, ranging from 1M to 100M vectors, includes rich, task-specific labels and evaluation metrics, enabling assessment of retrieval algorithms within the full application pipeline rather than in isolation. Iceberg benchmarks 13 state-of-the-art VSS methods and re-ranks them based on application-level metrics, revealing substantial deviations from traditional rankings derived purely from recall-latency evaluations. Building on these insights, we define a set of task-centric meta-features and derive an interpretable decision tree to guide practitioners in selecting and tuning VSS methods for their specific workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。