评测大模型在多人对话中的记忆能力,发现现有系统表现远未达标。
GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations

- 构建多角色对话数据集,模拟真实群聊中的角色互动与信息流转。
- 测试显示最强系统平均准确率仅46.0%,知识更新任务低至27.1%。
- 适合研究多用户协作、对话记忆与角色认知的AI开发者参考。
大型语言模型代理日益作为个人助理和工作伙伴,在长时对话中依赖记忆系统来提取、检索并应用信息。然而,现有记忆系统和评估基准均基于一对一单用户场景,而实际部署常涉及多人参与的群组与频道。这种脱节导致三个群组记忆特性未被测量:(i) 超出简单拼接对话的群组动态;(ii) 说话人对齐信念追踪,需为每位用户建模记忆;(iii) 观众适配语言,即心智理论引发的角色化词汇变化。我们提出GroupMemBench,全面暴露上述三类挑战。通过图结构合成管道生成具有可控回复结构的多角色对话,每条消息基于用户人格与目标受众设定。对抗式查询管道将每个问题绑定特定提问者,涵盖六类任务:多跳推理、知识更新、术语歧义、隐含推理、时间推理与拒绝回答,通过迭代搜索生成高难度且真实的查询,全面检验记忆能力。对主流记忆系统的评测显示显著性能下降:最强系统平均准确率仅46.0%,知识更新任务为27.1%,术语歧义任务为37.7%;而简单BM25基线反而优于多数代理记忆系统。这表明当前记忆摄入机制抹除了群组记忆所依赖的结构与语义特征,多用户记忆问题仍未解决。
原文摘要 · Abstract (English)
Large Language Model (LLM) agents increasingly serve as personal assistants and workplace collaborators, where their utility depends on memory systems that extract, retrieve, and apply information across long-running conversations. However, both existing memory systems and benchmarks are built around the dyadic, single-user setup, even though real deployments routinely span groups and channels with multiple users interacting with the agent and with each other. This mismatch leaves three properties of group memory unmeasured: (i) group dynamics that go beyond concatenated one-on-one chats, (ii) speaker-grounded belief tracking, where the per-user memory modeling is needed, and (iii) audience-adapted language, where Theory-of-Mind shifts produce role-specific vocabulary. We introduce GroupMemBench, a benchmark that exposes all three. A graph-grounded synthesis pipeline produces multi-party conversations with controllable reply structure and conditions each message on per-user personas and target audiences. An adversarial query pipeline then binds every question to a specific asker across six categories, spanning multi-hop reasoning, knowledge update, term ambiguity, user-implicit reasoning, temporal reasoning, and abstention, and iteratively searches challenging, realistic queries that reflect comprehensive memory capability. Benchmarking leading memory systems exposes a sharp collapse: the strongest one reaches only 46.0% average accuracy, with knowledge update at 27.1% and term ambiguity at 37.7%, while a simple BM25 baseline matches or exceeds most agent memory systems. This indicates current memory ingestion erases the structural and lexical features group memory depends on, leaving multi-user memory far from solved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。