提出因果对齐框架,让视频问答更准确识别关键画面。
Cross-modal Causal Relation Alignment for Video Question Grounding
- 用高斯平滑与双向对比学习减少跨模态虚假关联。
- 在两个数据集上提升视频定位精度,显著增强推理鲁棒性。
- 适合需要可靠视觉理解的多模态任务研究者使用。
视频问答(VideoQG)要求模型在回答问题的同时,推断支持答案的相关视频片段。然而,现有方法常受跨模态虚假相关性影响,难以准确识别与问题意图一致的主导视觉场景。此外,视觉-语言模型在挑战性下游任务如VideoQG中表现出不忠实的泛化能力与鲁棒性不足。本文提出一种新框架Cross-modal Causal Relation Alignment (CRA),以消除虚假相关并提升问答与视频时序定位之间的因果一致性。CRA包含三个核心组件:i) 高斯平滑定位(GSG)模块通过跨模态注意力估计时间区间,并利用自适应高斯滤波去噪;ii) 跨模态对齐(CMA)通过双向对比学习增强弱监督下的VideoQG性能;iii) 显式因果干预(ECI)模块实现多模态去混淆,对视觉采用前门干预、对语言采用后门干预。在两个VideoQG数据集上的大量实验表明,CRA能更有效地发现视觉相关的语义内容,并实现稳健的问题推理。代码已开源:https://github.com/WissingChen/CRA-GQA。
原文摘要 · Abstract (English)
Video question grounding (VideoQG) requires models to answer the questions and simultaneously infer the relevant video segments to support the answers. However, existing VideoQG methods usually suffer from spurious cross-modal correlations, leading to a failure to identify the dominant visual scenes that align with the intended question. Moreover, vision-language models exhibit unfaithful generalization performance and lack robustness on challenging downstream tasks such as VideoQG. In this work, we propose a novel VideoQG framework named Cross-modal Causal Relation Alignment (CRA), to eliminate spurious correlations and improve the causal consistency between question-answering and video temporal grounding. Our CRA involves three essential components: i) Gaussian Smoothing Grounding (GSG) module for estimating the time interval via cross-modal attention, which is de-noised by an adaptive Gaussian filter, ii) Cross-Modal Alignment (CMA) enhances the performance of weakly supervised VideoQG by leveraging bidirectional contrastive learning between estimated video segments and QA features, iii) Explicit Causal Intervention (ECI) module for multimodal deconfounding, which involves front-door intervention for vision and back-door intervention for language. Extensive experiments on two VideoQG datasets demonstrate the superiority of our CRA in discovering visually grounded content and achieving robust question reasoning. Codes are available at https://github.com/WissingChen/CRA-GQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。