测试发现大模型检索器在科研任务中不如传统方法，提出新框架提升检索效果。
SAGE: Benchmarking and Improving Retrieval for Deep Research Agents
- 用大模型生成关键词子查询，但效果不如传统检索器
- 传统BM25比LLM检索器高30%准确率，因现有系统生成关键词不精准
- 提出文档增强框架，让现成检索器性能提升8%和2%
深度研究代理系统在应对复杂科学问题方面展现出强大能力。与此同时，基于大语言模型（LLM）的检索器在指令遵循和推理方面也表现出色。这引出一个关键问题：基于大模型的检索器能否有效支持深度研究代理的工作流程？为此，我们提出了SAGE，一个涵盖四个科学领域的1,200个查询、包含20万篇论文的科学文献检索基准。我们评估了六种深度研究代理，发现所有系统在推理密集型检索任务中表现不佳。以DR Tulu为骨干模型，进一步对比了BM25与基于LLM的检索器（即ReasonIR和gte-Qwen2-7B-instruct）。令人惊讶的是，BM25性能比基于大模型的检索器高出约30%，原因在于现有代理生成的子查询过于依赖关键词。为提升性能，我们提出一种基于语料库的测试时扩展框架，利用大模型为文档添加元数据和关键词，使现成检索器更易使用。该方法在短答案和开放性问题上分别取得8%和2%的性能提升。
原文摘要 · Abstract (English)
Deep research agents have emerged as powerful systems for addressing complex queries. Meanwhile, LLM-based retrievers have demonstrated strong capability in following instructions or reasoning. This raises a critical question: can LLM-based retrievers effectively contribute to deep research agent workflows? To investigate this, we introduce SAGE, a benchmark for scientific literature retrieval comprising 1,200 queries across four scientific domains, with a 200,000 paper retrieval corpus. We evaluate six deep research agents and find that all systems struggle with reasoning-intensive retrieval. Using DR Tulu as backbone, we further compare BM25 and LLM-based retrievers (i.e., ReasonIR and gte-Qwen2-7B-instruct) as alternative search tools. Surprisingly, BM25 significantly outperforms LLM-based retrievers by approximately 30%, as existing agents generate keyword-oriented sub-queries. To improve performance, we propose a corpus-level test-time scaling framework that uses LLMs to augment documents with metadata and keywords, making retrieval easier for off-the-shelf retrievers. This yields 8% and 2% gains on short-form and open-ended questions, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。