首个荷兰语零样本信息检索基准，推动低资源语言模型发展
BEIR-NL: Zero-shot Information Retrieval Benchmark for the Dutch Language
- 将英文BEIR数据集自动翻译为荷兰语，构建BEIR-NL基准
- BM25在荷兰语上仍具竞争力，结合重排序后媲美最优稠密模型
- 翻译降低数据质量，提示跨语言迁移的局限性，适合多语言研究者
零样本信息检索模型常通过BEIR基准评估，该基准由多个异构数据集组成，覆盖不同领域与任务。但其全英文内容限制了对荷兰语等低资源语言的应用。为此，我们通过自动翻译公开的BEIR数据集，构建了首个荷兰语零样本检索基准BEIR-NL。基于此，我们评估了多种多语言稠密排序与重排序模型及词项方法BM25。实验表明，BM25仍是有力基线，仅被大型检索训练稠密模型超越；结合重排序后，性能可媲美最佳稠密模型。进一步通过反向翻译验证，发现翻译导致性能下降，说明翻译在构建基准时存在局限。BEIR-NL已开源至Hugging Face。
原文摘要 · Abstract (English)
Zero-shot evaluation of information retrieval (IR) models is often performed using BEIR; a large and heterogeneous benchmark composed of multiple datasets, covering different retrieval tasks across various domains. Although BEIR has become a standard benchmark for the zero-shot setup, its exclusively English content reduces its utility for underrepresented languages in IR, including Dutch. To address this limitation and encourage the development of Dutch IR models, we introduce BEIR-NL by automatically translating the publicly accessible BEIR datasets into Dutch. Using BEIR-NL, we evaluated a wide range of multilingual dense ranking and reranking models, as well as the lexical BM25 method. Our experiments show that BM25 remains a competitive baseline, and is only outperformed by the larger dense models trained for retrieval. When combined with reranking models, BM25 achieves performance on par with the best dense ranking models. In addition, we explored the impact of translation on the data by back-translating a selection of datasets to English, and observed a performance drop for both dense and lexical methods, indicating the limitations of translation for creating benchmarks. BEIR-NL is publicly available on the Hugging Face hub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。