arXiv:2412.09230cs.CVcs.AI2024-12被引 1

提出新方法,让视频理解更懂问题。

Foundation Models and Adaptive Feature Selection: A Synergistic Approach to Video Question Answering

  • 用跨注意力选关键帧,结合图结构捕捉物体动态。
  • 通过问答感知的动态图变换器,生成精细视频表征。
  • 在多个数据集上表现领先,适合视觉问答研究者。

本文针对视频问答(VideoQA)中的复杂挑战,提出一种新型框架LGQAVE,旨在更好融合问题与视频帧、语义对象级抽象,构建问答感知的视频表征。传统方法依赖随机抽帧,而本方法利用跨注意力机制精准定位与问题相关的视频帧,并通过独立图结构捕捉帧内物体动态,借助miniGPT模型将图结构与问题语义对齐。这些图由问答感知的动态图变换器(Q-DGT)处理,生成局部与全局视频表征。最后,通过交叉注意力模块融合局部与全局嵌入,形成最终视频表示,交由语言模型生成答案。在多个基准测试上的实验表明,LGQAVE在多项选择题和开放式问答任务中显著优于现有模型。

原文摘要 · Abstract (English)

This paper tackles the intricate challenge of video question-answering (VideoQA). Despite notable progress, current methods fall short of effectively integrating questions with video frames and semantic object-level abstractions to create question-aware video representations. We introduce Local-Global Question Aware Video Embedding (LGQAVE), which incorporates three major innovations to integrate multi-modal knowledge better and emphasize semantic visual concepts relevant to specific questions. LGQAVE moves beyond traditional ad-hoc frame sampling by utilizing a cross-attention mechanism that precisely identifies the most relevant frames concerning the questions. It captures the dynamics of objects within these frames using distinct graphs, grounding them in question semantics with the miniGPT model. These graphs are processed by a question-aware dynamic graph transformer (Q-DGT), which refines the outputs to develop nuanced global and local video representations. An additional cross-attention module integrates these local and global embeddings to generate the final video embeddings, which a language model uses to generate answers. Extensive evaluations across multiple benchmarks demonstrate that LGQAVE significantly outperforms existing models in delivering accurate multi-choice and open-ended answers.

视频问答多模态图神经网络注意力机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。