构建多语言RAG评估基准，测试嵌入模型语义理解能力
IRSC: A Zero-shot Evaluation Benchmark for Information Retrieval through Semantic Comprehension in Retrieval-Augmented Generation Scenarios
- 设计五类检索任务，覆盖查询、标题、段落等不同粒度
- 提出SSCI和RCCI新指标，量化模型语义理解与检索能力
- 揭示主流模型在跨语言场景下的性能短板，适合模型开发者参考
在使用大语言模型的检索增强生成（RAG）任务中，检索信息的质量对最终输出至关重要。本文提出了IRSC基准，用于评估嵌入模型在多语言RAG任务中的表现。该基准包含五种检索任务：查询检索、标题检索、段落内部分检索、关键词检索和摘要检索。针对当前嵌入模型在RAG场景下缺乏全面评测与有效比较方法的问题，本文引入了两种新指标：语义理解相似性指数（SSCI）和检索能力竞赛指数（RCCI），并评估了Snowflake-Arctic、BGE、GTE和M3E等模型。主要贡献包括：1）构建IRSC基准；2）提出SSCI与RCCI指标；3）揭示嵌入模型在跨语言场景中的局限性。该基准旨在提升对RAG中准确检索系统的设计与理解。所有代码与数据集已公开于：https://github.com/Jasaxion/IRSC_Benchmark
原文摘要 · Abstract (English)
In Retrieval-Augmented Generation (RAG) tasks using Large Language Models (LLMs), the quality of retrieved information is critical to the final output. This paper introduces the IRSC benchmark for evaluating the performance of embedding models in multilingual RAG tasks. The benchmark encompasses five retrieval tasks: query retrieval, title retrieval, part-of-paragraph retrieval, keyword retrieval, and summary retrieval. Our research addresses the current lack of comprehensive testing and effective comparison methods for embedding models in RAG scenarios. We introduced new metrics: the Similarity of Semantic Comprehension Index (SSCI) and the Retrieval Capability Contest Index (RCCI), and evaluated models such as Snowflake-Arctic, BGE, GTE, and M3E. Our contributions include: 1) the IRSC benchmark, 2) the SSCI and RCCI metrics, and 3) insights into the cross-lingual limitations of embedding models. The IRSC benchmark aims to enhance the understanding and development of accurate retrieval systems in RAG tasks. All code and datasets are available at: https://github.com/Jasaxion/IRSC_Benchmark
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。