用图结构记忆视频，让模型能跨帧追踪物体并回答复杂问题。
RAVU: Retrieval Augmented Video Understanding with Compositional Reasoning over Graph
- 构建时空图表示视频，把物体和动作关系存为长期记忆。
- 仅用5-10帧检索就超越当前最优方法，在两个数据集上表现更优。
- 适合需要多跳推理、跨帧跟踪的长视频理解任务。
理解长视频仍是大模态模型（LMMs）的重大挑战。现有LMMs因缺乏显式记忆与检索机制，难以处理数分钟至数小时的视频。为此，我们提出RAVU（Retrieval Augmented Video Understanding），一种通过在时空图上进行组合推理来增强视频理解的新框架。我们构建视频的图表示，捕捉实体间的时空关系，该图作为长期记忆，可追踪物体及其行为随时间的变化。针对复杂问题，我们将查询分解为一系列推理步骤，并在图上执行，检索相关关键信息。该方法显著提升对长视频的理解能力，尤其适用于需多跳推理和跨帧追踪的问题。在两个主流视频问答数据集NExT-QA和EgoSchema上，本方法仅使用5-10帧检索即达到优于其他SOTA方法和基线的表现。
原文摘要 · Abstract (English)
Comprehending long videos remains a significant challenge for Large Multi-modal Models (LMMs). Current LMMs struggle to process even minutes to hours videos due to their lack of explicit memory and retrieval mechanisms. To address this limitation, we propose RAVU (Retrieval Augmented Video Understanding), a novel framework for video understanding enhanced by retrieval with compositional reasoning over a spatio-temporal graph. We construct a graph representation of the video, capturing both spatial and temporal relationships between entities. This graph serves as a long-term memory, allowing us to track objects and their actions across time. To answer complex queries, we decompose the queries into a sequence of reasoning steps and execute these steps on the graph, retrieving relevant key information. Our approach enables more accurate understanding of long videos, particularly for queries that require multi-hop reasoning and tracking objects across frames. Our approach demonstrate superior performances with limited retrieved frames (5-10) compared with other SOTA methods and baselines on two major video QA datasets, NExT-QA and EgoSchema.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。