探索多层表示提升文档检索效果，比现有方法更准更快。
Investigating Multi-layer Representations for Dense Passage Retrieval
- 用编码器多层输出融合文档表示，捕捉更丰富的语言信息。
- 在单向量检索中超越双塔、ME-BERT和ColBERT，性能显著提升。
- 兼容先进训练技巧，适合追求高精度的检索系统研发者。
密集检索模型通常使用文档编码器最后一层隐藏层的向量表示文档，但这与预训练语言模型不同层的表示包含不同语言知识、微调行为各异的事实相悖。为此，我们提出利用多层编码器输出构建文档表示，称为多层表示（MLR）。首先研究不同层表示在多向量检索设置下的影响，随后提出池化策略将多向量模型简化为单向量模型以提升检索效率。实验表明，MLR在单向量检索设置下优于双塔模型、ME-BERT和ColBERT，且与检索导向预训练、难例挖掘等先进训练技术兼容良好。
原文摘要 · Abstract (English)
Dense retrieval models usually adopt vectors from the last hidden layer of the document encoder to represent a document, which is in contrast to the fact that representations in different layers of a pre-trained language model usually contain different kinds of linguistic knowledge, and behave differently during fine-tuning. Therefore, we propose to investigate utilizing representations from multiple encoder layers to make up the representation of a document, which we denote Multi-layer Representations (MLR). We first investigate how representations in different layers affect MLR's performance under the multi-vector retrieval setting, and then propose to leverage pooling strategies to reduce multi-vector models to single-vector ones to improve retrieval efficiency. Experiments demonstrate the effectiveness of MLR over dual encoder, ME-BERT and ColBERT in the single-vector retrieval setting, as well as demonstrate that it works well with other advanced training techniques such as retrieval-oriented pre-training and hard negative mining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。