构建真实长期记忆的评估基准，测试大模型在多应用中持续追踪用户变化的能力。
DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settings
- 基于15个月跨16个应用的合成数据，模拟用户属性、习惯、偏好随时间演变
- 系统随历史增长而记忆退化，93%失败源于记忆检索而非模型生成
- 揭示现有系统无法同时保留不变事实与更新变化信息，尤其在偏好判断上差
大型语言模型代理日益充当个人助手，需长期记住用户档案——包括身份属性、日常习惯和偏好，并随工作、生活节奏和口味变化而更新。现有基准通过简短、简化交互评估记忆能力，忽略了真实行为的三大核心特征：档案内容异质，属性、习惯与偏好以不同时间尺度演化；变化由外部上下文（如季节、人生事件）驱动；证据极少明确陈述，而是分散于各应用中的微小行为中，需系统自行推断。我们提出DynamicMem，一个合成基准，为每位用户构建长达15个月的活动数据，涵盖2.2M tokens平均文本量与1,772个有依据事件，覆盖电商、健身、社交等16个应用。用户档案在此期间持续演化，从不直接给出，每个属性、习惯或偏好都需从跨应用的细小信号中推断。我们在五个季度节点进行评估，跟踪系统随历史增长的表现。对五种代表性系统的基准测试揭示了单一准确率无法体现的问题：(i) 随历史延长，档案重建质量下降，但服务任务准确率保持平稳，尽管二者共享同一记忆；(ii) 无一系统能同时保留恒定事实并替换变化事实，错误集中于偏好判断和指代对象识别；(iii) 超过93%的失败源于记忆检索，而非模型生成答案，表明改进空间主要在记忆机制本身。
原文摘要 · Abstract (English)
LLM agents increasingly act as personal assistants that must remember a user's profile over months: who they are (attributes), what they routinely do (habits), and what they prefer (preferences), and keep it updated as jobs, routines, and tastes drift. Existing benchmarks evaluate this "memory" ability through short, simplified interactions, missing three core properties of real behavior: the profile is heterogeneous, with attributes, habits, and preferences evolving on different timelines; changes are driven by external context such as seasons and life events; and evidence is rarely stated explicitly, instead scattered across many small actions in different apps that a memory system must infer from. We introduce DynamicMem, a synthetic benchmark that constructs 15 months of activity per user, providing long-term multi-app data that real users' privacy keeps out of reach. It provides user-consistent trajectories averaging 2.2M tokens and 1,772 grounded events per user across 16 applications such as e-commerce, fitness, and social platforms. The profile evolves over this period and is never given explicitly: each attribute, habit, or preference must be inferred from small signals scattered across apps. We evaluate at five quarterly checkpoints to track how systems scale as history grows. Benchmarking five representative systems exposes problems a single accuracy score hides: (i) profile reconstruction degrades with history length while service-task accuracy stays flat, despite both drawing on the same memory; (ii) no system both keeps facts that stay true and replaces facts that change, with errors clustering on preferences and on naming the exact referent; and (iii) over 93% of failures trace to what the memory retrieves, not to the model writing the answer, so the largest room for improvement lies in memory itself. Code: https://wenyaxie023.github.io/DynamicMem/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。