破解高维向量检索的维度诅咒,揭示稳定性的关键作用
Breaking the Curse of Dimensionality: On the Stability of Modern Vector Retrieval
- 从稳定性视角重新分析高维检索,发现小扰动下邻居不变性是核心
- 实证验证三种常见检索场景的稳定性:分段距离保稳定,过滤惩罚可诱导稳定
- 为向量数据库设计提供理论指导,避免维度灾难导致的性能崩溃
现代向量数据库在高维神经嵌入上实现高效检索,支撑网页搜索到检索增强生成等应用。然而经典理论预测此类任务会受维度诅咒影响,即点间距离趋于一致,使最近邻搜索失效。本文从稳定性角度重新审视该矛盾:小查询扰动不改变最近邻。我们拓展稳定性理论至三种实际常用检索场景:(i) 多向量检索中,证明常用的Chamfer距离保持单向量稳定性,而平均池化聚合可能破坏它;(ii) 过滤向量检索中,表明足够大的不匹配过滤惩罚可诱导稳定性,即使底层搜索本身不稳定;(iii) 稀疏向量检索中,提出并证明新的充分稳定性条件。在合成与真实数据集上的实验结果与理论预测一致,为模型与系统设计提供了避免维度诅咒的具体指导。
原文摘要 · Abstract (English)
Modern vector databases enable efficient retrieval over high-dimensional neural embeddings, powering applications from web search to retrieval-augmented generation. However, classical theory predicts such tasks should suffer from the curse of dimensionality, where distances between points become nearly indistinguishable, thereby crippling efficient nearest-neighbor search. We revisit this paradox through the lens of stability, the property that small perturbations to a query do not radically alter its nearest neighbors. Building on foundational results, we extend stability theory to three key retrieval settings widely used in practice: (i) multi-vector search, where we prove that the popular Chamfer distance metric preserves single-vector stability, while average pooling aggregation may destroy it; (ii) filtered vector search, where we show that sufficiently large penalties for mismatched filters can induce stability even when the underlying search is unstable; and (iii) sparse vector search, where we formalize and prove novel sufficient stability conditions. Across synthetic and real datasets, our experimental results match our theoretical predictions, offering concrete guidance for model and system design to avoid the curse of dimensionality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。