生成式检索超越稠密检索,但对标识符歧义敏感。
Generative Retrieval Overcomes Limitations of Dense Retrieval but Struggles with Identifier Ambiguity
- 用合成数据集验证生成式检索直接预测相关性。
- 无需训练即达0.92~0.99的召回率,远超稠密检索。
- 解码机制缺陷导致标识符歧义,影响实际应用。
稠密检索模型虽流行,但在某些场景下表现不佳。生成式检索通过语言模型直接预测查询-文档相关性成为替代方案。本文使用名为LIMIT的合成数据集,该数据集曾用于揭示嵌入式检索的理论局限,但未用于评估生成式检索。我们填补空白,发现生成式检索在无额外训练下即取得优异性能(SEAL和MINDER的R@2分别为0.92和0.99),显著优于稠密检索(<0.03 R@2)和BM25(0.86 R@2)。但当在原数据集添加简单难负样本后,所有模型性能下降,包括生成式模型(R@2降至0.51)和BM25(降至0.21)。错误分析表明,解码机制失败源于无法生成唯一对应相关文档的标识符。未来工作需改进标识符设计或优化解码与打分算法以保留相关性信号。
原文摘要 · Abstract (English)
While dense retrieval models, which embed queries and documents into a shared low-dimensional space, have gained widespread popularity, they were shown to exhibit important theoretical limitations and considerably lag behind traditional sparse retrieval models in certain settings. Generative retrieval has emerged as an alternative approach to dense retrieval by using a language model to predict query-document relevance directly. In this paper, we demonstrate strengths and weaknesses of generative retrieval approaches using a simple synthetic dataset, called LIMIT, that was previously introduced to empirically demonstrate the theoretical limitations of embedding-based retrieval but was not used to evaluate generative retrieval. We close this research gap and show that generative retrieval achieves the best performance on this dataset without any additional training required (0.92 and 0.99 R@2 for SEAL and MINDER, respectively), compared to dense approaches (< 0.03 Recall@2) and BM25 (0.86 R@2). However, we then proceed to extend the original LIMIT dataset by adding simple hard negative samples and observe the performance degrading for all the models including the generative retrieval models (0.51 R@2) as well as BM25 (0.21 R@2). Error analysis identifies a failure in the decoding mechanism, caused by the inability to produce identifiers that are unique to relevant documents. Future generative retrieval must address these issues, either by designing identifiers that are more suitable to the decoding process or by adapting decoding and scoring algorithms to preserve relevance signals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。