用大模型优化问题,精准定位视频关键片段,提升复杂问答准确率。
LeAdQA: LLM-Driven Context-Aware Temporal Grounding for Video Question Answering
- 用大模型重写问题,消除因果歧义,聚焦时间重点。
- 在NExT-QA等数据集上达到当前最优表现,复杂推理准确率显著提升。
- 适合需要高精度视频理解与复杂逻辑推理的研究者使用。
视频问答(VideoQA)需在长视频中识别稀疏的关键时刻,并推理其因果关系以回答语义复杂的提问。现有方法受限于两种普遍但根本性缺陷:(1)任务无关采样盲目处理所有帧,使关键事件被无关内容淹没;(2)启发式检索仅捕捉表面模式,忽略复杂推理所需的因果-时序结构。为此,我们提出LeAdQA,通过融合因果感知的问题重构与细粒度视觉定位,弥合上述差距。首先利用大模型重构问题-选项对,消除因果歧义并强化时间聚焦;随后,优化后的问题引导时序定位模型精确检索最显著片段,并通过自适应融合机制动态整合证据以最大化相关性。最终,融合的视觉-文本线索由多模态大模型生成准确、上下文相关的答案。在NExT-QA、IntentQA和NExT-GQA上的实验表明,该方法的精准视觉定位显著提升了视频-问题关系的理解,在复杂推理任务上实现当前最优性能,同时保持计算高效。
原文摘要 · Abstract (English)
Video Question Answering (VideoQA) requires identifying sparse critical moments in long videos and reasoning about their causal relationships to answer semantically complex questions. While recent advances in multimodal learning have improved alignment and fusion, current approaches remain limited by two prevalent but fundamentally flawed strategies: (1) task-agnostic sampling indiscriminately processes all frames, overwhelming key events with irrelevant content; and (2) heuristic retrieval captures superficial patterns but misses causal-temporal structures needed for complex reasoning. To address these challenges, we introduce LeAdQA, an innovative approach that bridges these gaps through synergizing causal-aware query refinement with fine-grained visual grounding. Our method first leverages LLMs to reformulate question-option pairs, resolving causal ambiguities and sharpening temporal focus. These refined queries subsequently direct a temporal grounding model to precisely retrieve the most salient segments, complemented by an adaptive fusion mechanism dynamically integrating the evidence to maximize relevance. The integrated visual-textual cues are then processed by an MLLM to generate accurate, contextually-grounded answers. Experiments on NExT-QA, IntentQA, and NExT-GQA demonstrate that our method's precise visual grounding substantially enhances the understanding of video-question relationships, achieving state-of-the-art (SOTA) performance on complex reasoning tasks while maintaining computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。