构建首个代理搜索数据集,解决智能体查询与传统检索系统的不匹配问题。
A Picture of Agentic Search
- 设计新方法收集智能体在检索增强任务中的完整交互数据
- 发布ASQ数据集,包含3种代理在3个数据集上的查询与推理记录
- 适合研究智能体检索、下一代信息检索系统的设计者
随着自动化系统越来越多地与人类共同发起搜索请求,信息检索(IR)面临重大转变。然而,当前的IR体系仍以人类为中心,其系统设计、评估指标、用户模型和数据集均基于人类查询行为。这导致现有假设不再适用,工作负载量、可预测性及查询模式已发生变化,影响系统性能与优化:缓存效率下降,查询预处理可能增加开销却未提升效果,标准指标也可能误判满意度。若不适应,检索模型将无法满足人类或新兴的智能体用户。但目前缺乏捕捉代理搜索行为的数据集,而这对依赖数据驱动评估与优化的IR领域至关重要。本文提出一种方法,收集智能体检索增强系统在回答查询时产生的全部数据,并发布Agentic Search Queryset(ASQ)数据集。ASQ涵盖HotpotQA、Researchy Questions和MS MARCO中的推理生成查询、检索文档及思维过程,覆盖3类智能体与2种检索管道。配套工具包支持扩展至新代理、检索器和数据集。
原文摘要 · Abstract (English)
With automated systems increasingly issuing search queries alongside humans, Information Retrieval (IR) faces a major shift. Yet IR remains human-centred, with systems, evaluation metrics, user models, and datasets designed around human queries and behaviours. Consequently, IR operates under assumptions that no longer hold in practice, with changes to workload volumes, predictability, and querying behaviours. This misalignment affects system performance and optimisation: caching may lose effectiveness, query pre-processing may add overhead without improving results, and standard metrics may mismeasure satisfaction. Without adaptation, retrieval models risk satisfying neither humans, nor the emerging user segment of agents. However, datasets capturing agent search behaviour are lacking, which is a critical gap given IR's historical reliance on data-driven evaluation and optimisation. We develop a methodology for collecting all the data produced and consumed by agentic retrieval-augmented systems when answering queries, and we release the Agentic Search Queryset (ASQ) dataset. ASQ contains reasoning-induced queries, retrieved documents, and thoughts for queries in HotpotQA, Researchy Questions, and MS MARCO, for 3 diverse agents and 2 retrieval pipelines. The accompanying toolkit enables ASQ to be extended to new agents, retrievers, and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。