自动构建测试框架，评估大模型记忆使用能力
Minerva: A Programmable Memory Test Benchmark for Language Models
- 自动生成多样测试用例，覆盖搜索、回忆、编辑等原子任务
- 扩展传统测试范围，包含复合任务与结构化数据操作
- 提供可解释的评估结果，帮助定位模型记忆短板
大语言模型助手如何有效利用上下文记忆完成各类任务？传统数据集多为人工构造，存在静态、易过拟合、难解释且缺乏行动洞察等问题，无法准确识别模型失败的具体能力缺陷。本文提出一种自动生成全面测试用例的框架，用于评估模型对上下文记忆的有效利用能力。该框架拓展了现有研究中以（寻钥、键值、针在草堆）为主的核心搜索任务，涵盖搜索、回忆、编辑、匹配、比较等原子操作，以及将输入分块处理时的基础运算和状态维持能力，模拟真实数据场景。同时设计复合测试，考察模型执行更复杂集成任务的能力。本基准支持可解释、细致的模型记忆能力评估。
原文摘要 · Abstract (English)
How effectively can LLM-based AI assistants utilize their memory (context) to perform various tasks? Traditional data benchmarks, which are often manually crafted, suffer from several limitations: they are static, susceptible to overfitting, difficult to interpret, and lack actionable insights--failing to pinpoint the specific capabilities a model lacks when it does not pass a test. In this paper, we present a framework for automatically generating a comprehensive set of tests to evaluate models' abilities to use their memory effectively. Our framework extends the range of capability tests beyond the commonly explored (passkey, key-value, needle in the haystack) search, a dominant focus in the literature. Specifically, we evaluate models on atomic tasks such as searching, recalling, editing, matching, comparing information in context memory, performing basic operations when inputs are structured into distinct blocks, and maintaining state while operating on memory, simulating real-world data. Additionally, we design composite tests to investigate the models' ability to perform more complex, integrated tasks. Our benchmark enables an interpretable, detailed assessment of memory capabilities of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。