arXiv:2409.06299cs.CVcs.AI2024-09被引 18

通过分层事件记忆机制,提升长视频理解能力。

Enhancing Long Video Understanding via Hierarchical Event-Based Memory

论文配图:Enhancing Long Video Understanding via Hierarchical Event-Based Memory
图 1 · 摘自论文原文
  • 按事件自适应分割长视频,分别建模内部上下文。
  • 将前一事件信息压缩注入当前事件,增强跨事件依赖。
  • 适合需要精准理解长视频内容的研究与应用。

近期,将视觉基础模型与大语言模型(LLMs)结合以构建视频理解系统受到广泛关注。现有模型通常将整个视频的多样语义信息进行粗粒度压缩后输入LLM进行内容理解。该方法在短视频理解中表现良好,但在长视频中因压缩过于粗略,导致多个事件信息混合,产生冗余,关键事件语义被掩盖,削弱模型理解能力。为此,本文提出分层事件记忆增强型大语言模型(HEM-LLM),以更好理解长视频。首先设计一种新颖的自适应序列分割方案,将长视频中的多个事件分开处理;在此基础上对每个事件独立建模,建立事件内上下文关联,减少信息冗余。其次,在建模当前事件时,压缩并注入前一事件的信息,增强视频中长程跨事件依赖。最后,我们在多种视频理解任务上进行了广泛实验,结果表明本模型达到先进水平。

原文摘要 · Abstract (English)

Recently, integrating visual foundation models into large language models (LLMs) to form video understanding systems has attracted widespread attention. Most of the existing models compress diverse semantic information within the whole video and feed it into LLMs for content comprehension. While this method excels in short video understanding, it may result in a blend of multiple event information in long videos due to coarse compression, which causes information redundancy. Consequently, the semantics of key events might be obscured within the vast information that hinders the model's understanding capabilities. To address this issue, we propose a Hierarchical Event-based Memory-enhanced LLM (HEM-LLM) for better understanding of long videos. Firstly, we design a novel adaptive sequence segmentation scheme to divide multiple events within long videos. In this way, we can perform individual memory modeling for each event to establish intra-event contextual connections, thereby reducing information redundancy. Secondly, while modeling current event, we compress and inject the information of the previous event to enhance the long-term inter-event dependencies in videos. Finally, we perform extensive experiments on various video understanding tasks and the results show that our model achieves state-of-the-art performances.

视频理解长视频事件建模大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。