评测企业级复杂数据的深度搜索能力，揭示现有模型检索短板
Benchmarking Deep Search over Heterogeneous Enterprise Data
- 构建模拟企业流程的合成数据集，涵盖文档、聊天记录等多源异构数据
- 顶尖智能体检索增强生成系统平均得分仅32.96，远未达标
- 核心瓶颈是跨源多跳检索，适合评估长上下文大模型与RAG系统
我们提出一个新基准，用于评估深度搜索——一种需要在多样、稀疏但相关来源间进行源感知、多跳推理的真实复杂检索增强生成（RAG）任务。这些来源包括文档、会议记录、Slack消息、GitHub内容和网址，结构各异且常含人与人互动。我们通过合成数据管道模拟产品规划、开发和支持阶段的企业工作流，生成具有真实噪声和保证答案的多跳问题。基准包含39,190个企业资料作为检索池，支持可回答与不可回答查询，实现对长上下文大模型和RAG系统的细粒度评估。实验表明，即使最佳代理式RAG方法在该基准上平均得分仅为32.96。进一步分析显示，检索是主要瓶颈：现有方法难以完成深度搜索并获取全部必要证据，常基于不完整上下文推理，导致性能显著下降。
原文摘要 · Abstract (English)
We present a new benchmark for evaluating Deep Search--a realistic and complex form of retrieval-augmented generation (RAG) that requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources. These include documents, meeting transcripts, Slack messages, GitHub, and URLs, which vary in structure and often contain human-to-human interactions. We build it using a synthetic data pipeline that simulates business workflows across product planning, development, and support stages, generating interconnected content with realistic noise and multi-hop questions with guaranteed ground-truth answers. We release our benchmark with both answerable and unanswerable queries, and retrieval pool of 39,190 enterprise artifacts, enabling fine-grained evaluation of long-context LLM and RAG systems. Our experiments reveal that even the best-performing agentic RAG methods achieve an average performance score of 32.96 on our benchmark. With further analysis, we highlight retrieval as the main bottleneck: existing methods struggle to conduct deep searches and retrieve all necessary evidence. Consequently, they often reason over partial context, leading to significant performance degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。