用谱方法压缩文档向量，提升检索效率且不损失精度
EigenLI: Spectral Approximations to Late Interaction
- 通过分析发现晚交互表示具低秩特性，利用特征方向构建压缩空间
- 32维以内压缩后，比聚类方法在多个模型上表现更优
- 生成兼容近邻搜索的单向量表示，适用于高效检索系统
晚交互模型如ColBERT通过多标记级向量表示文档，虽效果优异但带来高索引成本、存储开销和昂贵的MaxSim计算。我们发现此类表示具有内在低秩结构：文档标记嵌入集中于一个低维子空间，能保留大部分检索信号。基于此，提出EigenLI谱近似框架，通过每个文档特有的低维子空间压缩表示。与聚类或池化方法不同，EigenLI识别文档的主导特征方向并用于构建降维交互表示。实验表明，当k≤32时，k-EigenLI在ColBERTv2和AnswerAI-ColBERT-small上优于k均值和Ward聚类池化方法；而GTE-ModernColBERT在k=32时聚类方法表现更佳。相同谱构造还生成EigenLI-SV，一种基于降维结构二阶摘要的ANN兼容单向量表示。在多个数据集及三种文本模型上，EigenLI-SV始终优于类似单向量替代方案如MUVERA。
原文摘要 · Abstract (English)
Late-interaction models such as ColBERT achieve strong effectiveness by representing each document with many token-level vectors, but this expressivity leads to large indexing cost, storage footprints and expensive MaxSim scoring. We show that late-interaction representations exhibit an intrinsic low-rank structure: document token embeddings concentrate in a low-dimensional subspace that preserves most of the retrieval signal. Leveraging this observation, we introduce EigenLI, a spectral approximation framework that compresses late-interaction representations via document-specific low-dimensional subspaces. Unlike clustering or pooling methods, EigenLI identifies the dominant eigendirections of each document and uses them to construct reduced interaction representations. Empirically, $k$-EigenLI with $k \le 32$ outperforms k-means and Ward clustering based pooling methods on ColBERTv2 and AnswerAI-ColBERT-small; GTE-ModernColBERT exhibits a different tradeoff at $k=32$, where clustering methods perform better. The same spectral construction also yields EigenLI-SV, an ANN-compatible single-vector representation derived from the second-order summary of the reduced structure. Across multiple datasets and all three text models, EigenLI-SV consistently outperforms comparable single-vector surrogates such as MUVERA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。