让AI理解视频中隐含意义,突破仅靠可见画面答题的局限。
Looking Beyond Visible Cues: Implicit Video Question Answering via Dual-Clue Reasoning
- 构建双线索推理框架,融合动作与意图线索进行隐性推断。
- 在隐式问答任务上超越GPT-4o等模型,提升最高达4.87%。
- 适合研究视频深层语义理解、广告意图分析的学者与工程师。
视频问答(VideoQA)旨在基于视频回答自然语言问题,以往工作主要依赖可直接观测的视觉证据(显性线索)。然而当问题涉及符号含义或深层意图时,显性线索常不可用,导致性能显著下降。为此,本文提出新任务与数据集I-VQA(隐式视频问答),聚焦于显性线索缺失场景下的问答。给定隐式问题与对应视频,I-VQA需依据视频中的上下文视觉线索进行推理。为此,我们提出隐式推理模型IRM,采用双流结构建模上下文动作与意图线索作为隐性推理链。IRM包含动作-意图模块(AIM)与视觉增强模块(VEM):AIM通过生成线索候选并进行关系推理来捕捉与问题相关的双重线索;VEM则利用关键上下文线索增强视觉表征。大量实验验证了IRM的有效性,在I-VQA任务上优于GPT-4o、OpenAI-o3和微调后的VideoChat2,分别提升0.76%、1.37%和4.87%。此外,IRM在类似隐式广告理解与交通场景未来预测任务中也达到当前最优表现。数据集与代码已公开于匿名仓库:https://github.com/tychen-SJTU/Implicit-VideoQA。
原文摘要 · Abstract (English)
Video Question Answering (VideoQA) aims to answer natural language questions based on the given video, with prior work primarily focusing on identifying the duration of relevant segments, referred to as explicit visual evidence. However, explicit visual evidence is not always directly available, particularly when questions target symbolic meanings or deeper intentions, leading to significant performance degradation. To fill this gap, we introduce a novel task and dataset, $\textbf{I}$mplicit $\textbf{V}$ideo $\textbf{Q}$uestion $\textbf{A}$nswering (I-VQA), which focuses on answering questions in scenarios where explicit visual evidence is inaccessible. Given an implicit question and its corresponding video, I-VQA requires answering based on the contextual visual cues present within the video. To tackle I-VQA, we propose a novel reasoning framework, IRM (Implicit Reasoning Model), incorporating dual-stream modeling of contextual actions and intent clues as implicit reasoning chains. IRM comprises the Action-Intent Module (AIM) and the Visual Enhancement Module (VEM). AIM deduces and preserves question-related dual clues by generating clue candidates and performing relation deduction. VEM enhances contextual visual representation by leveraging key contextual clues. Extensive experiments validate the effectiveness of our IRM in I-VQA tasks, outperforming GPT-4o, OpenAI-o3, and fine-tuned VideoChat2 by $0.76\%$, $1.37\%$, and $4.87\%$, respectively. Additionally, IRM performs SOTA on similar implicit advertisement understanding and future prediction in traffic-VQA. Datasets and codes are available for double-blind review in anonymous repo: https://github.com/tychen-SJTU/Implicit-VideoQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。