通过文档链接提升零样本检索的合成查询质量
Link, Synthesize, Retrieve: Universal Document Linking for Zero-Shot Information Retrieval
- 用相似度与实体识别构建文档链接，优化合成查询生成
- 在多个数据集上超越现有方法，零样本检索效果显著
- 适用于新领域、新语言的快速部署，适合工业级应用
尽管信息检索（IR）近期取得进展，零样本检索仍面临挑战，尤其在缺乏历史用户查询数据的新领域、新语言及新场景下。传统方法通常通过查询增强并基于合成查询对预训练模型进行微调。本文提出一种通用文档链接（UDL）算法，通过链接相似文档来提升跨不同特征数据集的合成查询生成能力。UDL利用熵选择相似度模型，并结合命名实体识别（NER）与相似度分数判断文档链接。实验表明，UDL在多种数据集和检索模型上均表现优异，显著优于现有最先进方法。代码已开源，确保可复现性。
原文摘要 · Abstract (English)
Despite the recent advancements in information retrieval (IR), zero-shot IR remains a significant challenge, especially when dealing with new domains, languages, and newly-released use cases that lack historical query traffic from existing users. For such cases, it is common to use query augmentations followed by fine-tuning pre-trained models on the document data paired with synthetic queries. In this work, we propose a novel Universal Document Linking (UDL) algorithm, which links similar documents to enhance synthetic query generation across multiple datasets with different characteristics. UDL leverages entropy for the choice of similarity models and named entity recognition (NER) for the link decision of documents using similarity scores. Our empirical studies demonstrate the effectiveness and universality of the UDL across diverse datasets and IR models, surpassing state-of-the-art methods in zero-shot cases. The developed code for reproducibility is included in https://github.com/eoduself/UDL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。