arXiv:2602.01313cs.CLcs.AI2026-02KDD被引 11

首个评估多方协作对话长期记忆的基准,真实模拟复杂社交场景。

Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues

  • 基于百万级跨话题多角色对话构建动态记忆评测框架
  • 现有模型在多角色推理中准确率仅26%,时间与角色感知能力严重不足
  • 适合研究长时记忆、社会交互与群体智能的学者使用

实际应用中的长时对话记忆本质上是协作性的:信息由多方产生,分散于多个群组和渠道,随时间演变,且隐含角色与社会语境。然而目前尚无基准能评估接近真实部署场景下的记忆表现,因现有评测主要聚焦二人或单一话题对话。本文提出 EverMemBench,首个面向长时协作记忆的基准,基于跨越百万词、包含密集跨话题交错、时间演进决策及角色条件人格的多方多群组对话数据构建。该基准通过 2400 个问答对,在细粒度召回、记忆意识和用户画像理解三个维度评估记忆系统。评估揭示当前系统存在根本性缺陷:即使拥有理想证据,多跳推理在多方归因下准确率仍低至 26%;缺乏显式版本语义时,时间推理失效;记忆意识受检索瓶颈制约,相似性方法会遗漏隐含相关项。EverMemBench 为实现真实世界记忆评估迈出了关键一步,是推动下一代能够基于时间、角色与协作结构推理的 LLM 发展的核心基准。代码与数据已公开于 https://github.com/EverMind-AI/EverMemBench。

原文摘要 · Abstract (English)

Long-term conversational memory in practical LLM applications is inherently collaborative: information is produced by multiple participants, scattered across groups and channels, revised over time, and implicitly grounded in roles and social context. Yet there is currently no established benchmark that evaluates memory under interaction patterns resembling real-world deployment, as existing benchmarks largely focus on dyadic or single-topic dialogues. In this paper, we introduce EverMemBench, the first benchmark designed for long-horizon collaborative memory, built from multi-party, multi-group conversations spanning over one million tokens with dense cross-topic interleaving, temporally evolving decisions, and role-conditioned personas. EverMemBench evaluates memory systems using 2400 QA pairs across three dimensions essential for real applications: fine-grained recall, memory awareness, and user profile understanding. Our evaluation reveals fundamental limitations of current systems: multi-hop reasoning collapses under multi-party attribution even with oracle evidence (26% accuracy), temporal reasoning fails without explicit version semantics beyond timestamps, and memory awareness is bottlenecked by retrieval, as similarity-based methods miss implicitly relevant information. EverMemBench thus represents a concrete step toward realistic evaluation of LLM memory and a cornerstone benchmark for developing next-generation LLMs that reason over time, roles, and collaborative interaction structure. Our benchmark and code are publicly available at https://github.com/EverMind-AI/EverMemBench.

对话记忆长时推理多角色协作

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。