arXiv:2608.20317cs.IR2026-08

将搜索基准的证据迁移到更大更真实的语料库,提升评估真实性。

Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search

  • 把原数据集的问答分解为可验证推理步骤,逐个在新语料中定位证据。
  • 迁移后保留57个完整有据可查的问题,检索召回率从84.3%降至21.4%。
  • 适用于任何可分解为事实的评测,适合研究真实场景下智能体搜索能力。

BrowseComp-Plus 基准通过使用固定语料库替代模糊的网络搜索,分离了智能体与检索器的角色。然而该语料库仅含约10万文档,且基于自身查询的支持文档和挖掘的难负样本构建,证据与干扰项均按查询选择。我们提出 $ ext{BrowseComp-Plus}_{ ext{CM}}$,保留原问题但将其证据迁移至 ClimbMix——NVIDIA 发布的 400B token、553M 文档的网页文本混合语料库,用于预训练语言模型,且未参考任何评测。核心贡献是投影流程:将每个问题分解为原子推理步骤,并在新语料中锚定每一步,仅当自动验证、独立智能体和人工审核均确认所有步骤有支持时才保留问题。该流程与数据集无关,适用于可分解为可验证事实的任何评测。应用于 830 个 BrowseComp-Plus 测试问题,共获得 57 个完全有据可查的问题及问题级相关性判断。迁移使检索难度显著上升:最强智能体答案准确率下降 5 分,证据召回率从 84.3% 降至 21.4%,同时搜索调用次数增加 63%。作为系列投影首篇,我们公开了流程、基准及分析结果于 https://github.com/castorini/cmass。

原文摘要 · Abstract (English)

The BrowseComp-Plus benchmark disentangled the evaluation of agentic search by replacing opaque web search with a fixed corpus, so that an agent's role can be separated from the retriever's. That corpus, however, holds only about 100K documents and was assembled from the supporting documents of the benchmark's own queries plus mined hard negatives, so the evidence and the distractors were both selected per query. We introduce $\text{BrowseComp-Plus}_{\text{CM}}$, which keeps the BrowseComp-Plus questions but relocates their evidence to ClimbMix, a 400B-token, 553M-document mixture of web text released by NVIDIA for pre-training language models and built without reference to any benchmark. Our main contribution is the projection pipeline that makes this possible: it decomposes each question into atomic reasoning hops and grounds every hop in the new corpus, retaining a question only when automatic verification, an independent agent, and human review all confirm that every hop is supported. The pipeline is dataset-agnostic and applies to any benchmark whose questions decompose into verifiable facts. Applied to the 830 BrowseComp-Plus test questions, our pipeline yields 57 fully grounded questions with question-level relevance judgments. Projection shifts the difficulty onto retrieval, as the strongest agent we evaluate loses five points of answer accuracy but sees its evidence recall fall from 84.3% to 21.4% while issuing 63% more search calls. As the first of a series of projections, we release the pipeline, the benchmark, and our analyses at https://github.com/castorini/cmass.

智能体搜索语料迁移评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。