防止向量库检索枢纽失控，通过哨兵查询实时监控文档准入。
Coverage Is Not Redundancy: Maintenance Cost and Exposure of Query-Aware Admission Indexes in Vector Databases Under Workload Drift
- 用哨兵查询检测文档是否成为检索枢纽，未达标则拒绝插入。
- 真实数据下召回率低于0.5时，控制失效，内存压缩索引问题更严重。
- 提出新检测机制，在0.1%插入开销内实现稳定防护，适合大规模部署。
在服务生产级检索的向量数据库中，单个文档可能被异常大量查询使用，形成检索枢纽并主导某主题结果。现有防御机制在插入时通过哨兵查询检查：仅当文档对哨兵的反kNN计数低于阈值τ时才允许插入。但随工作负载漂移，哨兵集作为在线维护的查询感知辅助索引，其维护成本影响插入性能。研究发现结构性限制——覆盖不等于冗余：一旦区域被覆盖，哨兵不再增加，但只有当τ个哨兵观测到枢纽时才会拒绝，导致暴露存在观测下限，无法通过降低更新或执行延迟消除。在880万向量的MS MARCO语料上测试HNSW、IVF-Flat和IVF-PQ索引，该下限仅为理想情况；随着索引召回率下降，暴露与数据翻动显著上升，召回率低于0.5时门控完全失效——尤其在百亿级使用的内存压缩型IVF-PQ上表现最差。引入一种召回感知的探测机制可恢复控制，代价为固定O(|S|d)，不足0.1%的近似最近邻插入开销。在真实新冠疫情工作负载漂移下验证该规律，在PostgreSQL/pgvector中实现0.33%插入税的门控，并将理论边界转化为按新兴区域配置哨兵预算的资源配置规则。计数测试能有效遏制枢纽，而检索时的归一化方法（NNN、QB-Norm）不能；预注册因果套件隔离出缺失覆盖机制，与两种嵌入族（BGE-1024、E5-768）的检索碎片化无关。
原文摘要 · Abstract (English)
In a vector database serving production-scale retrieval, a single inserted document can be retrieved for an anomalously large share of the query workload -- a retrieval hub -- and dominate the evidence returned for an entire topic. An emerging defense guards against this at ingest with an admission check: it maintains a set of sentinel queries and admits a document only if its reverse-kNN count against them stays below a threshold tau. Under workload drift this sentinel set is a query-aware auxiliary index that must be maintained online, and we study the cost that maintenance imposes on the ingest path. We identify a structural limit -- coverage is not redundancy: a monitor stops promoting sentinels once a region is covered, but the predicate rejects a hub only once tau sentinels witness it, so exposure has an observation-limited floor that no reduction in update or enforcement latency can close. On real HNSW, IVF-Flat, and IVF-PQ indexes over an 8.8M-vector MS MARCO corpus this floor is only a best case: as index recall falls, exposure and churn rise above it, and below recall ~0.5 the gate stops containing altogether -- worst on the memory-compressed IVF-PQ used at billion scale -- while a recall-aware witness probe restores containment at a fixed O(|S|d) admission cost, under 0.1% of the ANN insert. We validate the law under real (COVID-19) workload drift, implement the gate in PostgreSQL/pgvector at a 0.33% ingest tax, and turn the bound into a provisioning rule that sizes the sentinel budget per emerging region. A count test contains the hub where retrieval-time score normalizers (NNN, QB-Norm) do not, and a pre-registered causal suite isolates the missing-coverage mechanism from retrieval fragmentation across two embedding families (BGE-1024, E5-768).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。