让视频问答模型真正理解问题意图，精准定位关键时间片段。
What You Ask is What You Ground: Bridging Question Intent to Temporal Evidence for Grounded VideoQA
- 用可学习通信令牌实现视觉与语言的定向交互，动态调整视频特征。
- 在NExT-GQA和STAR数据集上达到当前最优，显著提升不同问题间的定位差异性。
- 适合关注视频理解、多模态对齐与细粒度时序定位的研究者。
我们研究了基础视频问答中一个被忽视的关键缺陷：问题无关的定位，即模型对同一视频的不同问题给出几乎相同的时间片段。根源在于以往设计的两个结构性局限：(i) 模态隔离，即在接收问题语义前就固定视频表示；(ii) 问答信息在定位模块内注入过弱。为此，我们提出GroundFormer，通过可学习通信令牌，在定位前将视频特征基于问题意图进行条件化，实现定向跨模态交互。在此基础上，采用因子分解的MIL交叉注意力，在候选级别监督下联合完成答案选择与时间证据匹配；同时利用高斯平滑将尖锐注意力转化为时序连贯的片段。此外，引入分阶段多模态对比损失，通过双遍训练流程对齐视频、问题与答案嵌入。GroundFormer在NExT-GQA和STAR数据集上达到最先进性能，显著提升问题区分性的时序定位能力。
原文摘要 · Abstract (English)
We study a critical yet overlooked failure mode in Grounded Video Question Answering: question-invariant grounding, where models predict nearly identical temporal segments for different questions about the same video. We trace this behavior to two structural limitations in prior common designs: (i) modality isolation that fixes video representations before they receive question semantics, and (ii) weak question injection inside the grounding module. To address this, we propose GroundFormer, which conditions video features on question intent before localization via learnable communication tokens that mediate directed visuo-lingual interaction. On top of the question-conditioned features, a factorized MIL cross-attention couples answer selection with temporal evidence under candidate-level supervision, while Gaussian smoothing converts peaked attention into temporally coherent segments. We further introduce a hierarchical multi-modal contrastive loss that aligns video, question, and answer embeddings across a two-pass training pipeline. GroundFormer achieves state-of-the-art grounded VideoQA performance on NExT-GQA and STAR, substantially improving question-discriminative temporal grounding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。