arXiv:2607.16848cs.LGcs.AI2026-07

评测大模型从论文中恢复信息的能力,强调需控制检索预算和方式。

Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration

  • 提出两个全文本科学记忆数据集,评估模型在限定预算下还原论文内容的能力。
  • 发现当前排行榜结果受检索粒度、预算等协议影响,控制预算后领先优势消失。
  • 推荐以预算约束的上下文恢复为评价标准,适合研究型AI系统评估。

长时记忆正成为大模型智能体的核心,但现有基准多聚焦对话或摘要,而科研智能体需从完整论文中恢复证据。本文引入两个全文本科学记忆基准:Public AI Memory(PAIM,81篇论文,66个问题)和Public Transformers(PTr,252篇论文,98个问题)。评估八种记忆/检索系统,包括自研的Theoria及无检索基线。结果显示,记忆排行榜不可靠,因数据摄入粒度、原始文本保留、检索预算、模态、评分审计和评判者选择均影响结果。例如,PAIM中Graphiti胜出,但每查询使用260万字符检索内容;控制预算后领先消失。在PTr中,对可清洁集成BM25的系统,稀疏-稠密混合检索是最大改进,Simple RAG、Mem0和Theoria的混合版本在0.03分内并列第一。多评判者与人类对照显示,大模型作为评判者结果稳定,与人工评估一致,有效分辨力约1分(十分类尺度)。主张科学记忆应作为有预算、模态感知的上下文恢复来评估,并公开数据集、工具链、原始输出与脚本以支持复现与未来评估。代码地址:http://gitlab.com/quantellence/research/scientific-recall-bench,数据集:http://huggingface.co/datasets/quantellence/srb-data。

原文摘要 · Abstract (English)

Long-term memory is becoming a core component of LLM agents, but most memory benchmarks evaluate conversations or compact summaries, while research agents need to restore evidence from full scientific papers. We introduce two full-text scientific-memory benchmarks, Public AI Memory (PAIM; 81 papers, 66 questions) and Public Transformers (PTr; 252 papers, 98 questions). We evaluate eight memory/retrieval systems, including our own proposed Theoria, plus a no-retrieval baseline. Our results show that memory leaderboards are not interpretable without the full protocol: ingestion granularity, raw-text preservation, retrieval budget, retrieval modality, rubric audit, and judge choice all affect the outcome. For example, on PAIM Graphiti wins convincingly but uses 2.6M characters of retrieved context per query, and after controlling for retrieval budget the lead disappears. On PTr, for the systems where BM25 retrieval can be added cleanly, the sparse-dense hybrid is the single most significant intervention: hybrid variants of Simple RAG, Mem0, and Theoria tie for the lead within 0.03 points. Multi-judge and human side-by-side calibration show that LLM-as-a-judge rankings are consistent across frontier judges and agree with human evaluation, with an effective resolution of roughly one point on a ten-point scale. We argue that scientific memory should be evaluated as budgeted, modality-aware context restoration rather than as an unconstrained architecture leaderboard, and we release the datasets, harness, raw outputs, judgments, and scripts to reproduce our results and serve as tools for such evaluation. Our code is available at http://gitlab.com/quantellence/research/scientific-recall-bench , and the datasets are available at http://huggingface.co/datasets/quantellence/srb-data .

科学记忆检索评估预算控制上下文恢复

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。