为连续生活日志设计新评估基准,发现复杂记忆系统反而不如简单方案。
Evaluating Memory Capability in Continuous Lifelog Scenario

- 构建分层合成框架生成双子集基准数据集
- 在线评估协议避免时间泄露,模拟真实流式场景
- 复杂记忆模型表现不及简易RAG基线,凸显保真度重要性
可穿戴设备可持续记录环境对话,为记忆系统带来新机遇。但现有评测多聚焦于在线一对一聊天或人机交互,忽视真实场景需求。由于公开的生活日志音频数据集稀缺,我们提出一种分层合成框架,构建新型基准 extbf{ extsc{LifeDialBench}},包含两个互补子集:基于真实第一视角视频的 extbf{EgoMem} 与基于模拟虚拟社区构建的 extbf{LifeMem}。关键在于,为解决传统离线设置中的时间泄漏问题,我们提出 extbf{Online Evaluation} 协议,严格遵循时间因果性,确保系统以真实流式方式被评估。实验结果揭示一个反直觉现象:当前复杂的记忆系统未能超越简单的RAG基线。这凸显了过度设计结构与有损压缩对性能的负面影响,强调在生活日志场景中高保真上下文保留的必要性。
原文摘要 · Abstract (English)
Nowadays, wearable devices can continuously lifelog ambient conversations, creating substantial opportunities for memory systems. However, existing benchmarks primarily focus on online one-on-one chatting or human-AI interactions, thus neglecting the unique demands of real-world scenarios. Given the scarcity of public lifelogging audio datasets, we propose a hierarchical synthesis framework to curate \textbf{\textsc{LifeDialBench}}, a novel benchmark comprising two complementary subsets: \textbf{EgoMem}, built on real-world egocentric videos, and \textbf{LifeMem}, constructed using simulated virtual community. Crucially, to address the issue of temporal leakage in traditional offline settings, we propose an \textbf{Online Evaluation} protocol that strictly adheres to temporal causality, ensuring systems are evaluated in a realistic streaming fashion. Our experimental results reveal a counterintuitive finding: current sophisticated memory systems fail to outperform a simple RAG-based baseline. This highlights the detrimental impact of over-designed structures and lossy compression in current approaches, emphasizing the necessity of high-fidelity context preservation for lifelog scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。