arXiv:2409.20365cs.CV2024-09EMNLP被引 44

用LLM实现零样本长视频理解,聚焦关键时空信息。

VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMs

  • 基于事件与内容的时空推理,筛选关键信息。
  • 在三个基准上超越当前最佳,平均提升超10%。
  • 适合需要高效处理长视频的AI研究者。

在视频-语言领域,基于零样本大语言模型(LLM)推理的视频理解方法已具备与传统端到端模型竞争的能力。然而,长视频理解因时间跨度长、信息冗余严重,对零样本LLM推理仍构成挑战。本文提出VideoINSTA框架,即面向零样本长视频理解的可解释时空推理方法。该框架包含:(1)基于零样本的长视频理解框架;(2)以事件为单位的时序推理与基于内容的空间推理机制;(3)自反思式信息推理策略,根据信息充分性与预测置信度动态平衡时间因素。在EgoSchema、NextQA、IntentQA和ActivityNetQA四个长视频问答数据集上,模型显著优于现有方法,其中在IntentQA上提升达12.7%。代码已开源。

原文摘要 · Abstract (English)

In the video-language domain, recent works in leveraging zero-shot Large Language Model-based reasoning for video understanding have become competitive challengers to previous end-to-end models. However, long video understanding presents unique challenges due to the complexity of reasoning over extended timespans, even for zero-shot LLM-based approaches. The challenge of information redundancy in long videos prompts the question of what specific information is essential for large language models (LLMs) and how to leverage them for complex spatial-temporal reasoning in long-form video analysis. We propose a framework VideoINSTA, i.e. INformative Spatial-TemporAl Reasoning for zero-shot long-form video understanding. VideoINSTA contributes (1) a zero-shot framework for long video understanding using LLMs; (2) an event-based temporal reasoning and content-based spatial reasoning approach for LLMs to reason over spatial-temporal information in videos; (3) a self-reflective information reasoning scheme balancing temporal factors based on information sufficiency and prediction confidence. Our model significantly improves the state-of-the-art on three long video question-answering benchmarks: EgoSchema, NextQA, and IntentQA, and the open question answering dataset ActivityNetQA. The code is released here: https://github.com/mayhugotong/VideoINSTA.

视频理解LLM长视频时空推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。