基于俄语维基百科创建新检索数据集,支持事实核查与文档检索等任务。
Wikipedia-based Datasets in Russian Information Retrieval Benchmark RusBEIR
- 从俄语维基百科‘你知道吗’栏目的句子级标注中构建数据集
- 词法方法在全文检索中表现优于神经模型,神经模型更擅长短文本语义捕捉
- 数据集公开可用,适合俄语信息检索与多语言研究者使用
本文基于俄语维基百科的“你知道吗”栏目,构建了一套新型俄语信息检索数据集。该数据集支持事实核查、检索增强生成和全文档检索等多种任务,通过带等级相关性标注的句子级事实及其参考文章实现。我们描述了数据集创建方法,可扩展现有俄语信息检索资源。实验对比了词法检索模型(如BM25)与微调后的俄语及多语言神经架构,结果表明:词法方法在全文档检索中表现更优,而神经模型在短文本(如事实核查或细粒度检索)中更能捕捉语义。我们还分析了文档长度对检索性能的影响,并证明结合检索与神经重排序能持续提升效果。所有数据集已在HuggingFace公开,完整代码已开源以支持复现与后续研究。
原文摘要 · Abstract (English)
In this paper, we present a novel series of Russian information retrieval datasets constructed from the "Did you know..." section of Russian Wikipedia. Our datasets support a range of retrieval tasks, including fact-checking, retrieval-augmented generation, and full-document retrieval, by leveraging interesting facts and their referenced Wikipedia articles annotated at the sentence level with graded relevance. We describe the methodology for dataset creation that enables the expansion of existing Russian Information Retrieval (IR) resources. Through extensive experiments, we extend the RusBEIR research by comparing lexical retrieval models, such as BM25, with state-of-the-art neural architectures fine-tuned for Russian, as well as multilingual models. Results of our experiments show that lexical methods tend to outperform neural models on full-document retrieval, while neural approaches better capture lexical semantics in shorter texts, such as in fact-checking or fine-grained retrieval. Using our newly created datasets, we also analyze the impact of document length on retrieval performance and demonstrate that combining retrieval with neural reranking consistently improves results. Our contribution expands the resources available for Russian information retrieval research and highlights the importance of accurate evaluation of retrieval models to achieve optimal performance. All datasets are publicly available at HuggingFace. To facilitate reproducibility and future research, we also release the full implementation on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。