arXiv:2504.10739cs.MMeess.IV2025-04中稿 · CVPR被引 15

模仿海马体机制,实现长时视听事件的跨模态理解

HippoMM: Hippocampal-inspired Multimodal Memory for Long Audiovisual Event Understanding

  • 基于海马体原理分三步:分割输入、压缩记忆、分层检索
  • 在自建数据集上达78.2%准确率,速度比基线快5倍
  • 适合研究认知启发的多模态系统或长期记忆建模者

理解长时间的视听体验对计算系统仍具挑战,尤其在时间整合与跨模态关联方面,这正是人类情景记忆的核心。我们提出HippoMM,一种映射海马体机制的计算认知架构。该架构不依赖规模扩展或复杂结构,而是集成三个组件:(i) 情景分割通过检测视听输入变化将视频切分为离散片段,模拟齿状回的模式分离;(ii) 记忆巩固将片段压缩为保留关键特征的摘要,类比海马体记忆形成;(iii) 分层记忆检索先搜索语义摘要,再围绕种子片段按时间窗口扩展,实现跨模态查询,模仿CA3的模式完成。三者协同构建了超越部分之和的整体系统。在自建的HippoVlog基准测试中,HippoMM取得78.2%的关联记忆准确率,运行速度比检索增强基线快5倍。结果表明,认知架构可为下一代多模态理解提供蓝图。代码与数据集已公开于https://github.com/linyueqian/HippoMM。

原文摘要 · Abstract (English)

Comprehending extended audiovisual experiences remains challenging for computational systems, particularly temporal integration and cross-modal associations fundamental to human episodic memory. We introduce HippoMM, a computational cognitive architecture that maps hippocampal mechanisms to solve these challenges. Rather than relying on scaling or architectural sophistication, HippoMM implements three integrated components: (i) Episodic Segmentation detects audiovisual input changes to split videos into discrete episodes, mirroring dentate gyrus pattern separation; (ii) Memory Consolidation compresses episodes into summaries with key features preserved, analogous to hippocampal memory formation; and (iii) Hierarchical Memory Retrieval first searches semantic summaries, then escalates via temporal window expansion around seed segments for cross-modal queries, mimicking CA3 pattern completion. These components jointly create an integrated system exceeding the sum of its parts. On our HippoVlog benchmark testing associative memory, HippoMM achieves state-of-the-art 78.2% accuracy while operating 5x faster than retrieval-augmented baselines. Our results demonstrate that cognitive architectures provide blueprints for next-generation multimodal understanding. The code and benchmark dataset are publicly available at https://github.com/linyueqian/HippoMM.

多模态理解认知架构长时记忆海马体模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。