提出一种新型度量空间近邻搜索索引,用随机采样构建可扩展的倒排结构。
misi: a Metric Inverted Sample Index
- 用随机采样点作为词汇表,对象由其最近的k_b个采样点表示
- 查询通过加权共享邻居投票+精确验证,0.99召回率下验证开销随n^0.30增长
- 构建并行且确定,适合内存受限或频繁重建的数据场景
我们提出misi,一种用于一般度量空间近似最近邻搜索的倒排索引。其词汇表为数据库的随机采样,大小与数据量n成正比。每个对象由其k_b个最近采样点表示,通过可替换的内层索引在采样点上查找;查询通过idf加权的共享邻居投票,并对C个候选进行精确验证。该设计将NAPP索引从固定数量的轴心点推广到线性规模的词汇表,在n增长时保持帖子列表的期望长度ρ = k_b/α恒定,使任意高召回率索引在αn个点上的性能可直接拓展至n个点,适用于任意度量空间。概率模型提供召回保证:当k_b在n的对数级时,覆盖重叠间隙即可满足要求,且验证预算由索引自身估计;同时给出理论极限:投票无法分辨低于1/√k_b量级的重叠差异。该设计优势在于结构:构建过程为n次独立搜索,可并行、确定性,64核上处理10^8向量仅需5,250秒,比同召回率图结构快3.7倍;支持3 GiB内存限制下的流式构建,便携版本在8 GB内存内从NVMe服务10^8向量,低于SSD图基线的工作内存下限。代价是查询耗时:饱和图基线在内存中快6-16倍,0.99召回率下的验证预算随n^0.30增长。所有结果均含种子、饱和扫描和完整配置,来自运行清单生成,包含实测负结果。
原文摘要 · Abstract (English)
We present misi, an inverted index for approximate nearest-neighbor search over general metric spaces whose vocabulary is a random sample of the database, of size proportional to $n$. Each object is represented by its $k_b$ nearest sample points, found by a pluggable inner index over the sample; queries are answered by an idf-weighted shared-neighbor vote followed by exact verification of $C$ candidates. The construction generalizes the NAPP index from a constant number of pivots to a linear-size vocabulary, which keeps posting lists at constant expected length $ρ= k_b/α$ as $n$ grows and turns the index into a combinator: any high-recall index on $αn$ points yields an index on $n$ points, for any metric. A probabilistic model gives a recall guarantee -- $k_b$ logarithmic in $n$ over the overlap gap suffices, with a verification budget the index itself estimates -- and a matching limit: the vote cannot resolve overlap differences below order $1/\sqrt{k_b}$. The design's strengths are structural: construction is $n$ independent searches -- embarrassingly parallel, deterministic, $5{,}250$ s for $10^8$ vectors on 64 cores, $3.7\times$ faster than a matched-recall graph build -- it streams under an enforced 3 GiB cap, and the portable artifact serves $10^8$ vectors from NVMe within an enforced 8 GB budget, below the working floor of the SSD-graph baseline. Its cost is query-time work: saturated graph baselines answer $6$-$16\times$ faster in RAM, and the verification budget for 0.99 recall grows as $n^{0.30}$. All results carry seeds, saturation sweeps and full configurations, are generated from run manifests, and include measured negative results. The intended applications weight construction cost, determinism, memory footprint, or black-box metrics over peak throughput: frequently rebuilt corpora, batch similarity workloads, constrained-memory serving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。