发现稠密检索模型隐含可直接用于传统检索的稀疏词汇，无需额外训练。
Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies
- 从稠密检索模型中提取出符合齐夫定律的稀疏词汇
- 在不依赖任何监督的情况下实现与SPLADE相当甚至更优的性能
- 特别适合提升单向量检索失败场景下的表现
我们提出Latent Terms，揭示了经过训练的稠密检索模型（无论单向量或多向量）能够被轻易分解为可直接用于检索的稀疏特征。在冻结检索器上训练时，无需任何检索相关调整的稀疏自编码器即可提取出具有近似齐夫分布统计特性的潜在词汇表，可直接用于经典稀疏检索评分方法BM25。该方法无需学习扩展目标或稀疏检索监督，可应用于任意稠密检索器。Latent Terms在自身基础模型上达到或超越单向量评分方法，也优于同类SPLADE变体；在专为暴露单向量检索缺陷而设计的LIMIT任务上，表现显著优于其基础模型。结果表明，神经检索器蕴含比默认评分函数更丰富、可索引的结构，但其他方法仍可有效利用。
原文摘要 · Abstract (English)
We propose Latent Terms, a method revealing that models trained for dense retrieval, whether single- or multi-vector, learn representations that can trivially be decomposed into retrieval-ready sparse features. When trained on frozen retrievers, Sparse Autoencoders without any retrieval-specific adjustments extract a latent vocabulary with approximately Zipfian collection statistics, directly suitable for classical sparse retrieval scoring via BM25. This approach enables sparse retrieval while requiring no learned expansion objective or sparse retrieval supervision whatsoever, and can be readily applied to any dense retriever. Latent Terms is able to match or outperform single-vector scoring methods from its own base model as well as comparable SPLADE variants. In addition, it substantially outperforms its base model on LIMIT, a task specifically designed to highlight the failures of single-vector retrieval. Overall, our results highlight that neural retrievers contain more expressive and indexable structure than their default scoring functions expose, but that other methods can nonetheless be leveraged.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。