提出分层记忆架构,让视频智能体更高效地长期理解内容
From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents

- 构建感知-情景-符号三层记忆结构,逐步提炼视觉细节为语义要点
- 在4个基准上达到顶尖性能,长时视频任务准确率提升12.3%
- 适合需要长期记忆的视频分析、智能代理等场景
尽管多模态大语言模型在短时推理中表现优异,但在长时视频理解上受限于有限的上下文窗口和静态记忆机制,难以模拟人类认知效率。现有方法通常分为两类:以视觉为中心的方法因密集视觉积累导致高延迟与冗余;以文本为中心的方法则因过度摘要造成细节丢失与幻觉。为此,我们提出基于模糊痕迹理论的分层多模态记忆架构MM-Mem,将记忆分层为感官缓冲区、情景流与符号模式,实现从细粒度感知痕迹(verbatim)到高层语义模式(gist)的渐进式提炼。为进一步控制记忆构建过程,我们推导出语义信息瓶颈目标,并引入SIB-GRPO以优化记忆压缩与任务相关性之间的权衡。推理阶段设计熵驱动的自上而下记忆检索策略。在4个基准上的实验证明,MM-Mem在离线与流式任务中均达到当前最优表现,展现出强大泛化能力,验证了受认知启发的记忆组织有效性。代码与配置已公开于https://github.com/EliSpectre/MM-Mem。
原文摘要 · Abstract (English)
While multimodal large language models have demonstrated impressive short-term reasoning, they struggle with long-horizon video understanding due to limited context windows and static memory mechanisms that fail to mirror human cognitive efficiency. Existing paradigms typically fall into two extremes: vision-centric methods that incur high latency and redundancy through dense visual accumulation, or text-centric approaches that suffer from detail loss and hallucination via aggressive captioning. To bridge this gap, we propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory. MM-Mem structures memory hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic Schema, enabling the progressive distillation of fine-grained perceptual traces (verbatim) into high-level semantic schemas (gist). Furthermore, to govern the dynamic construction of memory, we derive a Semantic Information Bottleneck objective and introduce SIB-GRPO to optimize the trade-off between memory compression and task-relevant information retention. In inference, we design an entropy-driven top-down memory retrieval strategy. Extensive experiments across 4 benchmarks confirm that MM-Mem achieves state-of-the-art performance on both offline and streaming tasks, demonstrating robust generalization and validating the effectiveness of cognition-inspired memory organization. Code and associated configurations are publicly available at https://github.com/EliSpectre/MM-Mem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。