通过调整词汇表大小和预训练权重，提升稀疏检索模型的效率与效果。
The Role of Vocabularies in Learning Sparse Representations for Ranking
- 用10万词表构建BERT模型，对比预训练与随机初始化差异。
- 在计算预算下，10万词表模型优于3.2万词表基准模型。
- 词汇表配置影响语义表示，适合优化稀疏检索系统设计。
学习稀疏检索（LSR）如SPLADE因其高效的倒排索引特性，在语义第一阶段匹配中日益受到关注。最近有研究提出使用扩展词汇表的SPLADE（ESPLADE），将查询与文档映射到自定义粒度的稀疏空间。然而，目前对词汇表在SPLADE模型中的作用及其与检索效率、效果的关系尚缺乏深入研究。为此，我们构建了10万词表大小的BERT模型，一个基于ESPLADE预训练初始化，另一个随机初始化。在真实搜索点击日志上微调后，采用基于逻辑分数的查询与文档剪枝，以在计算预算内平衡效率。实验结果表明，在剪枝条件下，两个10万词表模型在BM25框架下的计算预算内均优于3.2万词表的常规SPLADE模型；且ESPLADE初始化模型比随机初始化模型更有效，同时保持相近的检索成本。结果表明，输出词汇表的规模与预训练权重在检索引擎中起到了配置查询、文档及其交互语义表征的关键作用，超越其原本在NLP中的意义。这些发现为通过词汇表配置优化稀疏检索的表示能力提供了新方向。
原文摘要 · Abstract (English)
Learned Sparse Retrieval (LSR) such as SPLADE has growing interest for effective semantic 1st stage matching while enjoying the efficiency of inverted indices. A recent work on learning SPLADE models with expanded vocabularies (ESPLADE) was proposed to represent queries and documents into a sparse space of custom vocabulary which have different levels of vocabularic granularity. Within this effort, however, there have not been many studies on the role of vocabulary in SPLADE models and their relationship to retrieval efficiency and effectiveness. To study this, we construct BERT models with 100K-sized output vocabularies, one initialized with the ESPLADE pretraining method and one initialized randomly. After finetune on real-world search click logs, we applied logit score-based queries and documents pruning to max size for further balancing efficiency. The experimental result in our evaluation set shows that, when pruning is applied, the two models are effective compared to the 32K-sized normal SPLADE model in the computational budget under the BM25. And the ESPLADE models are more effective than the random vocab model, while having a similar retrieval cost. The result indicates that the size and pretrained weight of output vocabularies play the role of configuring the representational specification for queries, documents, and their interactions in the retrieval engine, beyond their original meaning and purposes in NLP. These findings can provide a new room for improvement for LSR by identifying the importance of representational specification from vocabulary configuration for efficient and effective retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。