提出可分解流式记忆生命周期的测试框架,揭示大模型外存性能关键影响因素。
Neuromem: A Granular Decomposition of the Streaming Lifecycle in External Memory for LLMs
- 构建流式插入与查询交织的评测协议,分解记忆生命周期为五维度
- 内存结构决定性能上限,压缩与生成整合主要影响成本分布
- 随轮次增长性能普遍下降,时间类查询仍最困难,适合外存优化研究者
现有外部记忆模块评估多基于静态设定:记忆离线构建后以固定状态查询。实际中记忆呈流式:新信息持续到达,插入与检索交错进行,记忆状态在服务过程中动态演化。此场景下,准确率与成本由完整的记忆生命周期决定,涵盖信息的摄入、维护、检索及生成集成。本文提出Neuromem,一个可扩展的测试平台,在插入与检索交织的协议下基准化外部记忆模块,并将生命周期细分为五个维度:记忆数据结构、归一化策略、合并策略、查询构造方式与上下文集成机制。基于三个代表性数据集LOCOMO、LONGMEMEVAL和MEMORYAGENTBENCH,Neuromem在统一服务栈中评估可互换变体,报告分词级F1与插入/检索延迟。总体发现:性能通常随轮次增长而下降,时间相关查询仍是最具挑战类别。记忆数据结构显著决定可达到的质量边界,而激进压缩与生成集成机制仅在插入与检索间转移成本,带来有限准确率提升。
原文摘要 · Abstract (English)
Most evaluations of External Memory Module assume a static setting: memory is built offline and queried at a fixed state. In practice, memory is streaming: new facts arrive continuously, insertions interleave with retrievals, and the memory state evolves while the model is serving queries. In this regime, accuracy and cost are governed by the full memory lifecycle, which encompasses the ingestion, maintenance, retrieval, and integration of information into generation. We present Neuromem, a scalable testbed that benchmarks External Memory Modules under an interleaved insertion-and-retrieval protocol and decomposes its lifecycle into five dimensions including memory data structure, normalization strategy, consolidation policy, query formulation strategy, and context integration mechanism. Using three representative datasets LOCOMO, LONGMEMEVAL, and MEMORYAGENTBENCH, Neuromem evaluates interchangeable variants within a shared serving stack, reporting token-level F1 and insertion/retrieval latency. Overall, we observe that performance typically degrades as memory grows across rounds, and time-related queries remain the most challenging category. The memory data structure largely determines the attainable quality frontier, while aggressive compression and generative integration mechanisms mostly shift cost between insertion and retrieval with limited accuracy gain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。