让HNSW搜索既快又准,理论保证不丢精度
HNSW with Accuracy Guarantees Using Graph Spanners

- 用统计方法动态验证HNSW搜索质量,低质时自动转精确算法
- 通过几何图跨度理论估算最近邻最大距离,实现数学上界控制
- 兼顾速度与正确性,适合对结果可靠性有要求的场景
层级可导航小世界(HNSW)图因对数级复杂度和优异的实证性能成为行业标准。然而,其依赖贪婪遍历策略,缺乏理论正确性保障。本文提出一种‘先验证后修正’框架,首次在保持高效的同时提供理论保证。该框架首先使用无分布假设的统计验证器,以极低开销动态评估标准HNSW搜索结果质量;若检测到低质量结果,则安全切换至精确恢复算法。为使精确恢复可行,我们将HNSW重视为几何图跨度,并利用极端值理论随机估计其最大经验拉伸因子,从而数学上界定真实最近邻的最大距离。在多个基准数据集上的大量实验表明,该分层框架在平均情况下保持HNSW的速度,同时确保最坏情况下的精确性,优于其他可比方法。
原文摘要 · Abstract (English)
Hierarchical Navigable Small World (HNSW) graphs serve as the industry standard due to their logarithmic complexity and strong empirical performance. However, HNSW relies on greedy graph traversal, a heuristic that provides no theoretical guarantees of correctness. In this paper, we propose a novel "Certify-then-Rectify" framework that bridges the gap between the speed of heuristic search and the rigor of exact retrieval. Rather than discarding HNSW, our approach first employs a distribution-free statistical certifier to dynamically evaluate the quality of a standard HNSW search with minimal overhead. If certification indicates that the retrieved neighbors are of low quality, the framework safely escalates to a rigorous exact recovery algorithm. To make this exact recovery computationally feasible, we reinterpret the HNSW graph as a geometric spanner and utilize Extreme Value Theory to stochastically estimate its maximum empirical stretch factor. This allows us to mathematically bound the maximum distance of true nearest neighbors. Extensive evaluations on benchmark datasets demonstrate that our tiered framework delivers the average-case speed of HNSW while ensuring the worst-case correctness of exact search and outperforming other applicable approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。