生成式检索可突破密集检索的性能瓶颈,尤其在大规模数据和模型下表现更优。
Does Generative Retrieval Overcome the Limitations of Dense Retrieval?
- 通过直接生成文档标识符,改变传统检索的局部归一化优化方式。
- 在更大语料库和模型规模下,性能下降更少,且表征能力随参数增长而提升。
- 适合关注可扩展检索系统、大模型应用的研究者参考。
生成式检索(GR)作为神经信息检索的新范式,通过直接生成相关文档标识符,为密集检索(DR)提供替代方案。本文从理论与实证两方面分析了GR与DR在学习目标与表征能力上的根本差异:GR采用全局归一化的最大似然优化,将语料库与相关性信息直接编码于模型参数中;而DR采用局部归一化目标,依赖外部嵌入表示语料库,并通过双线性交互计算相似度。分析表明,在模型与数据规模扩大时,GR可克服DR的固有局限,带来两大优势:其一,面对更大语料库时,避免因局部归一化导致的优化漂移引发的性能骤降;其二,随着模型增大,其表征能力随参数量提升,不受限于制约DR的全局低秩结构。我们在Natural Questions与MS MARCO数据集上,通过控制负样本策略、嵌入维度与模型规模的实验验证了上述理论结论。然而,尽管具备理论优势,GR在实践中并未普遍优于DR。论文进一步指出弥合理论潜力与实际性能差距的方向,为未来可扩展、鲁棒的生成式检索研究提供指引。
原文摘要 · Abstract (English)
Generative retrieval (GR) has emerged as a new paradigm in neural information retrieval, offering an alternative to dense retrieval (DR) by directly generating identifiers of relevant documents. In this paper, we theoretically and empirically investigate how GR fundamentally diverges from DR in both learning objectives and representational capacity. GR performs globally normalized maximum-likelihood optimization and encodes corpus and relevance information directly in the model parameters, whereas DR adopts locally normalized objectives and represents the corpus with external embeddings before computing similarity via a bilinear interaction. Our analysis suggests that, under scaling, GR can overcome the inherent limitations of DR, yielding two major benefits. First, with larger corpora, GR avoids the sharp performance degradation caused by the optimization drift induced by DR's local normalization. Second, with larger models, GR's representational capacity scales with parameter size, unconstrained by the global low-rank structure that limits DR. We validate these theoretical insights through controlled experiments on the Natural Questions and MS MARCO datasets, across varying negative sampling strategies, embedding dimensions, and model scales. But despite its theoretical advantages, GR does not universally outperform DR in practice. We outline directions to bridge the gap between GR's theoretical potential and practical performance, providing guidance for future research in scalable and robust generative retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。