新基准评估对话系统如何整合分散信息,而非仅记忆事实。
UTILMEM: Benchmarking Evidence Utilization in Long-Term Conversational Memory

- 构建1717个实例的基准,测试跨会话推理与证据整合能力。
- 现有系统在事实回忆上表现好,但整合信息能力普遍薄弱。
- 适合研究长时对话记忆、多轮推理与抗干扰模型的学者。
长期记忆对对话智能体日益重要,但现有评测主要依赖点对点的事实召回:系统能否从过往交互中恢复孤立事实或事件细节。然而真实场景中的记忆使用更复杂:需将分散、隐含且嘈杂的证据,从长时间对话历史中整合为连贯、任务导向的输出。我们称此能力为记忆利用(memory utilization)。本文提出 UtilMem,一个包含1,717个样本、覆盖五个领域的诊断性基准,用于评估记忆利用的四个未被充分探索方面:对密集历史的推理、识别隐含相关记忆、将分散证据合成摘要、分析或计划,以及抵抗语义相似干扰项的能力。对多种基于检索与记忆增强的系统进行评估发现,传统事实记忆表现优异的系统,并不意味着具备有效记忆利用能力。此外,仅靠检索不足:即使成功获取相关证据,系统仍常无法跨会话整合信息,或区分有用证据与看似合理的干扰项。这些结果揭示了信息存取与有效使用之间的显著差距,表明长时对话记忆的进步需要显式支持证据整合与抗检索干扰的架构。代码已开源。
原文摘要 · Abstract (English)
Long-term memory is increasingly important for conversational agents, yet existing benchmarks primarily measure memory through pointwise factual recall: whether a system can recover isolated facts or event-level details from prior interactions. Real-world memory use, however, often requires a more demanding capability: integrating distributed, implicit, and noisy evidence across extended interaction histories into coherent, task-oriented outputs. We call this capability memory utilization. Here, we introduce UtilMem, a diagnostic benchmark comprising 1,717 instances across five domains, designed to evaluate four underexplored aspects of memory utilization: reasoning over dense histories, identifying implicitly relevant memories, synthesizing distributed evidence into summaries, analyses, or plans, and resisting interference from semantically similar distractors. Evaluating a diverse set of retrieval-based and memory-augmented systems, we find that strong performance on conventional factual-memory benchmarks does not reliably translate into effective memory utilization. Moreover, retrieval alone is insufficient: even when relevant evidence is successfully recovered, systems frequently fail to integrate information across sessions or to distinguish useful evidence from plausible distractors. These findings expose a substantial gap between accessing stored information and using it effectively, and suggest that progress in long-term conversational memory will require architectures that explicitly support evidence integration and robustness to retrieval interference. Code is available at https://github.com/peijunallin/UtilMem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。