构建可分离检索与推理能力的科研评估沙盒,揭示大模型真实推理水平。
Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities
- 通过四种实验范式解耦检索与推理,量化各自损失。
- 多模型测试显示部分模型在复杂信息下表现更差,存在模式切换脆弱性。
- 专为科研深度推理设计,适合评估模型真实科学理解能力。
尽管现有基准表现良好,大语言模型能否基于真正新颖的科学信息进行推理仍不明确。多数评估混杂了检索、工具链选择与参数记忆,且受开放网络波动影响。本文提出DeR2——一个可控的深度研究沙盒,通过四类实验范式(仅指令、黄金概念、仅相关文档、完整集合)将证据获取与推理分离,保留深度搜索的核心挑战:多步整合、去噪与基于证据的结论生成。为防止参数泄漏,采用两阶段验证:要求无证据时失败但能通过概念解题。每个任务均提供冻结文档库(2023–2025年理论论文),并附专家标注的概念与验证推理链。对多种主流基础模型的实验表明,性能差异显著,部分模型在完整集合下表现反而劣于仅指令情况,暴露出模式切换脆弱性和概念误用问题。
原文摘要 · Abstract (English)
Despite strong performance on existing benchmarks, it remains unclear whether large language models can reason over genuinely novel scientific information. Most evaluations score end-to-end RAG pipelines, where reasoning is confounded with retrieval and toolchain choices, and the signal is further contaminated by parametric memorization and open-web volatility. We introduce DeR2, a controlled deep-research sandbox that isolates document-grounded reasoning while preserving core difficulties of deep search: multi-step synthesis, denoising, and evidence-based conclusion making. DeR2 decouples evidence access from reasoning via four regimes--Instruction-only, Concepts (gold concepts without documents), Related-only (only relevant documents), and Full-set (relevant documents plus topically related distractors)--yielding interpretable regime gaps that operationalize retrieval loss vs. reasoning loss and enable fine-grained error attribution. To prevent parametric leakage, we apply a two-phase validation that requires parametric failure without evidence while ensuring oracle-concept solvability. To ensure reproducibility, each instance provides a frozen document library (drawn from 2023-2025 theoretical papers) with expert-annotated concepts and validated rationales. Experiments across a diverse set of state-of-the-art foundation models reveal substantial variation and significant headroom: some models exhibit mode-switch fragility, performing worse with the Full-set than with Instruction-only, while others show structural concept misuse, correctly naming concepts but failing to execute them as procedures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。