评估生成式检索模型在分布外场景下的鲁棒性，发现其泛化能力有待提升。
On the Robustness of Generative Information Retrieval Models
- 从查询变化、新查询类型等四方面构建分布外评估框架
- 生成式模型在新数据分布下表现显著弱于密集检索模型
- 为构建更可靠的检索系统提供实证依据，适合关注模型可靠性研究者
生成式信息检索通过直接生成文档标识符来召回文档，已有大量工作致力于提升其有效性。然而对其鲁棒性的关注仍不足。评估生成式IR模型在分布外（OOD）场景下的泛化能力至关重要，即模型在面对新数据分布时的表现如何？为此，本文从四个角度分析了检索任务中的OOD情形：(i) 查询变化；(ii) 未见过的查询类型；(iii) 未见过的任务；(iv) 语料库扩展。基于此分类体系，我们对代表性生成式IR模型与密集检索模型进行了实证对比研究。结果表明，生成式模型在分布外情况下的鲁棒性亟待改进。通过对生成式模型的分布外性能分析，本文旨在推动更可靠的信息检索模型的发展。代码已开源：https://github.com/Davion-Liu/GR_OOD。
原文摘要 · Abstract (English)
Generative information retrieval methods retrieve documents by directly generating their identifiers. Much effort has been devoted to developing effective generative IR models. Less attention has been paid to the robustness of these models. It is critical to assess the out-of-distribution (OOD) generalization of generative IR models, i.e., how would such models generalize to new distributions? To answer this question, we focus on OOD scenarios from four perspectives in retrieval problems: (i)query variations; (ii)unseen query types; (iii)unseen tasks; and (iv)corpus expansion. Based on this taxonomy, we conduct empirical studies to analyze the OOD robustness of representative generative IR models against dense retrieval models. Our empirical results indicate that the OOD robustness of generative IR models is in need of improvement. By inspecting the OOD robustness of generative IR models we aim to contribute to the development of more reliable IR models. The code is available at \url{https://github.com/Davion-Liu/GR_OOD}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。