构建多语言信息检索评测集，支持中、波、俄及英之间的跨语言检索测试。
NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval
- 整合2022至2024年TREC NeuCLIR任务数据，覆盖150个单语与跨语言查询
- 包含25万条相关性判断，支持英-中/波/俄多语言检索评估
- 提供神经排序模型融合基线，替代传统BM25作为第一阶段检索
为衡量检索技术进展，需具备能准确区分系统性能的相关性标注测试集。本文提出NeuCLIRBench，一个面向跨语言与多语言检索的评测集合。该集合包含中文、波斯语、俄语的原生文档，以及这些文档的机器翻译英文版本。支持多种检索场景：英语、中文、波斯语或俄语的单语检索；以英语为查询语言，其他三种语言之一为文档语言的跨语言检索；以及以英语为查询语言，涵盖所有三种语言文档的多语言检索。NeuCLIRBench整合了2022、2023、2024年TREC NeuCLIR赛道的主题。单语与跨语言任务约150个查询，共250,128条相关性判断；多语言检索任务100个查询。数据集提供强大统计辨别力以区分不同检索方法。集合附带强神经检索系统的融合基线，使重排算法开发者不再依赖BM25作为首阶段检索器。NeuCLIRBench已公开可用。
原文摘要 · Abstract (English)
To measure advances in retrieval, test collections with relevance judgments that can faithfully distinguish systems are required. This paper presents NeuCLIRBench, an evaluation collection for cross-language and multilingual retrieval. The collection consists of documents written natively in Chinese, Persian, and Russian, as well as those same documents machine translated into English. The collection supports several retrieval scenarios including: monolingual retrieval in English, Chinese, Persian, or Russian; cross-language retrieval with English as the query language and one of the other three languages as the document language; and multilingual retrieval, again with English as the query language and relevant documents in all three languages. NeuCLIRBench combines the TREC NeuCLIR track topics of 2022, 2023, and 2024. The 250,128 judgments across approximately 150 queries for the monolingual and cross-language tasks and 100 queries for multilingual retrieval provide strong statistical discriminatory power to distinguish retrieval approaches. A fusion baseline of strong neural retrieval systems is included with the collection so that developers of reranking algorithms are no longer reliant on BM25 as their first-stage retriever. NeuCLIRBench is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。