arXiv:2605.17415cs.LGcs.AI2026-05

无需校准的流式向量搜索新方法,稳定提升召回率。

IVF-TQ: Calibration-Free Streaming Vector Search via a Codebook-Free Residual Layer

  • 用固定随机旋转+预计算量化器替代可学习码本,实现数据无关压缩。
  • 在相同比特预算下,相比传统方法提升17.7个百分点召回率。
  • 无需重训练码本或调参,适用于多种数据集和存储场景。

针对流式数据中近似最近邻索引召回率随时间下降的问题,本文提出IVF-TQ,一种无需码本校准的倒排文件索引。其残差压缩层采用固定随机旋转与预计算的Lloyd-Max标量量化器,仅依赖比特数b和维度d,不随数据变化。理论证明了基于球面内积误差的结构化保证。该设计带来类似放大效应,在匹配比特预算下相较Flat TQ提升17.7pp召回率,且超越Extended RaBitQ至统计噪声水平。在九个受控实验(三组1000万数据、三类PQ内存配置、三组种子)中,传统方法每批次重训练码本无法恢复流式差距;而IVF-PQ需对每数据集调参,但IVF-TQ仅用一组固定参数(b, d)即在所有数据集上保持稳定,误差Δ在[-0.80, +0.56]pp之间。核心贡献在于:无需训练码本、无需调参、无需重训练周期即可闭合性能差距。

原文摘要 · Abstract (English)

Approximate nearest neighbor (ANN) indexes deployed against streaming corpora silently lose recall over weeks. The standard diagnosis is distribution shift, but under shuffled-i.i.d. ingestion -- no shift at all -- product quantization still degrades -3.8pp at sub-matched bit budgets. The dominant production compression methods (PQ, OPQ, ScaNN) all fit a codebook to an initial sample and reuse it as the database grows by orders of magnitude. This paper presents IVF-TQ, an inverted-file index whose residual compression layer is data-independent: a fixed random rotation followed by a precomputed Lloyd-Max scalar quantizer parameterised only by the bit width b and dimension d. Only the IVF coarse k-means partition is trained. A uniform-over-sphere inner-product error bound depending only on (b, d, delta) provides a structural guarantee no learned-codebook method admits. The same codebook-free design enables an IVF-amplification effect that closes the gap to Extended RaBitQ to within statistical noise (+17.7pp over flat TQ at matched bit budget), and an Adaptive variant that refreshes the partition without touching the compression layer. Across nine controlled cells (three 10M datasets, three PQ memory regimes, three seeds), per-batch PQ codebook retraining never recovers the streaming gap; IVF-PQ streaming stability requires per-dataset bit-budget tuning, while IVF-TQ holds at one fixed (b, d) configuration on all three datasets with Delta in [-0.80, +0.56]pp. The contribution is operational: no codebook to train, no per-dataset bit-budget tuning, no retraining cycle that ever closes the gap.

向量搜索流式处理无码本

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。