改进稀疏检索模型的关键词频率惩罚,显著降低生产环境检索延迟。
An Alternative to FLOPS Regularization to Effectively Productionize SPLADE-Doc
- 引入DF-FLOPS正则化,针对高词频项进行惩罚以缩短倒排索引列表。
- 使检索速度提升约10倍,且在跨领域任务中性能仅下降2.2点MRR@10。
- 适用于需高效部署的工业级搜索引擎,兼顾速度与召回效果。
学习型稀疏检索(LSR)模型将文本编码为加权词项向量,需保持稀疏性以利用倒排索引结构进行检索。SPLADE作为最流行的LSR模型,使用FLOPS正则化来训练期间促进向量稀疏性。然而,该方法仅确保查询或文档内部的稀疏性,不控制词项整体频率。高词频(DF)项在生产检索引擎(如Apache Solr)中因过长的词频列表导致显著延迟。为此,本文提出新型正则化技术DF-FLOPS,通过惩罚高DF项的使用,有效缩短词频列表并降低检索延迟。与推理时的停用词过滤等方法不同,DF-FLOPS可保留真正关键的高频词。实验表明,该方法成功减少了高DF项的出现,使检索速度提升约10倍,在域内任务中仅损失2.2点MRR@10,跨域任务中13项中有12项表现更优。最终检索延迟接近BM25水平,为LSR在生产环境中的部署提供了重要进展。
原文摘要 · Abstract (English)
Learned Sparse Retrieval (LSR) models encode text as weighted term vectors, which need to be sparse to leverage inverted index structures during retrieval. SPLADE, the most popular LSR model, uses FLOPS regularization to encourage vector sparsity during training. However, FLOPS regularization does not ensure sparsity among terms - only within a given query or document. Terms with very high Document Frequencies (DFs) substantially increase latency in production retrieval engines, such as Apache Solr, due to their lengthy posting lists. To address the issue of high DFs, we present a new variant of FLOPS regularization: DF-FLOPS. This new regularization technique penalizes the usage of high-DF terms, thereby shortening posting lists and reducing retrieval latency. Unlike other inference-time sparsification methods, such as stopword removal, DF-FLOPS regularization allows for the selective inclusion of high-frequency terms in cases where the terms are truly salient. We find that DF-FLOPS successfully reduces the prevalence of high-DF terms and lowers retrieval latency (around 10x faster) in a production-grade engine while maintaining effectiveness both in-domain (only a 2.2-point drop in MRR@10) and cross-domain (improved performance in 12 out of 13 tasks on which we tested). With retrieval latencies on par with BM25, this work provides an important step towards making LSR practical for deployment in production-grade search engines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。