构建首个大规模智能体检索评估基准，覆盖十种语言真实用户查询。
Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems
- 基于真实对话生成机器重写查询，匹配生产环境检索场景
- 1900万文档+7万条多语言查询，提供三类标注体系减少漏检
- 验证不同模型在跨领域/语言下的表现差异，支持快速采样评估
评估大规模生产环境中智能体检索系统的第一阶段检索器，需要一个将大规模语料库与大量由真实用户查询和对话历史重构的机器生成查询相匹配的基准，并为每个查询标注多个相关文档。现有公开基准无法满足此需求：大规模语料库通常仅提供少量测试查询，而拥有大量查询的基准文档量普遍不足千万级。此外，多数基准使用人工撰写的查询，而智能体RAG流水线中的第一阶段检索器处理的是与人类搜索行为分布不同的机器重写查询。为此，我们提出Q2D-Web（Query2Doc-Web），一个包含1.9亿文档和7万条十语言智能体重写查询的大规模检索基准，查询源自生产系统的真实用户查询。Q2D-Web提供三类固定相关性判断：智能体引用、生产排序结果，以及结合两者并用LLM补全未标注文档的联合集合。我们对13种检索器（包括词法、稠密及后期交互模型）进行评测，发现其相对排序对判断集选择不敏感，但在主题领域、查询语言和查询类型上表现差异显著。为实现快速评估，我们研究了子语料采样方法：保留约三分之一的语料，通过多模型检索结果的倒数排名融合选取，可使联合判断下的模型排序与全语料一致，同时召回率@1000仅下降3至7点。公开排行榜地址：https://huggingface.co/spaces/perplexity-ai/q2d-web-leaderboard
原文摘要 · Abstract (English)
Evaluating first-stage retrievers in large-scale production RAG requires a benchmark that pairs a large-scale corpus with a large set of agent-reformulated search queries based on real user queries and their conversation threads, and that labels many relevant documents per query. No existing public benchmark evaluates this setting: large-scale collections typically provide only a small number of evaluation queries, whereas benchmarks with many queries generally contain only millions of documents. Moreover, most benchmarks assess human-written queries, while the first-stage retrievers in agentic RAG pipelines serve machine-written reformulations whose distribution differs from human search behavior. To overcome these evaluation gaps, we introduce Q2D-Web (Query2Doc-Web), a large-scale agentic retrieval benchmark consisting of a 190M-document web corpus and 70k agentic search queries in ten languages, reformulated from real-world user queries in production systems. Q2D-Web provides three sets of fixed relevance judgments: agent citations, production rankings, and a combined set that unions both signals and adds LLM-based judgments of unlabeled pooled documents to reduce false negatives. We benchmark 13 retrievers including lexical, dense, and late-interaction models and find that their relative ordering is largely insensitive to the choice of judgment set, while diverging substantially across topical domains, query languages, and query types. To enable fast evaluation, we also study subcorpus sampling as an approximation to full-corpus evaluations. Retaining a third of the corpus, selected by reciprocal rank fusion over pooled retriever runs, preserves the full-corpus model ranking under the combined judgments while raising absolute Recall@1000 only by 3 to 7 points. The public leaderboard is accessible under: https://huggingface.co/spaces/perplexity-ai/q2d-web-leaderboard
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。