构建多跳检索数据集StratRAG，评测RAG系统在噪声环境下的推理能力。
StratRAG: A Multi-Hop Retrieval Evaluation Dataset for Retrieval-Augmented Generation Systems
- 基于HotpotQA构造三类问题，每题配15文档含2篇正确答案与13个干扰项。
- 混合检索方法表现最佳（Recall@2=0.70，MRR=0.93），桥接类问题仍难突破。
- 适合评估和改进RAG系统的多跳推理与抗干扰能力。
我们提出StratRAG，一个开源的多跳检索评估数据集，用于在真实、嘈杂的文档池条件下基准测试检索增强生成（RAG）系统的性能。该数据集源自HotpotQA（干扰项设置），包含2,200个样本，涵盖三类问题：桥接、比较和是非题。每个问题对应一组15个候选文档，其中恰好包含2个正确文档和13个主题相关但非正确的干扰项。我们对三种检索策略进行了基准测试：BM25、密集检索（all-MiniLM-L6-v2）以及混合融合，报告了验证集上的Recall@k、MRR和NDCG@5指标。混合检索整体表现最优（Recall@2 = 0.70，MRR = 0.93），但桥接类问题仍显著困难（Recall@2 = 0.67），提示未来需探索基于强化学习的检索策略。StratRAG已公开发布于 https://huggingface.co/datasets/Aryanp088/StratRAG。
原文摘要 · Abstract (English)
We introduce StratRAG, an open-source retrieval evaluation dataset for benchmarking Retrieval-Augmented Generation (RAG) systems on multi-hop reasoning tasks under realistic, noisy document-pool conditions. Derived from HotpotQA (distractor setting), StratRAG comprises 2,200 examples across three question types -- bridge, comparison, and yes-no -- each paired with a pool of 15 candidate documents containing exactly 2 gold documents and 13 topically related distractors. We benchmark three retrieval strategies -- BM25, dense retrieval (all-MiniLM-L6-v2), and hybrid fusion -- reporting Recall@k, MRR, and NDCG@5 on the validation set. Hybrid retrieval achieves the best overall performance (Recall@2 = 0.70, MRR = 0.93), yet bridge questions remain substantially harder (Recall@2 = 0.67), motivating future work on reinforcement-learning-based retrieval policies. StratRAG is publicly available at https://huggingface.co/datasets/Aryanp088/StratRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。