arXiv:2507.04958cs.CVcs.MM2025-07中稿 · ACM MM 2025被引 4

用因果推理消除文本与视频的虚假关联,提升时序句子定位鲁棒性。

Boosting Temporal Sentence Grounding via Causal Inference

  • 构建因果模型,通过干预文本变量消除语义偏见。
  • 在公开数据集上显著提升定位准确率,尤其在分布外样本上表现更优。
  • 适合关注模型可解释性与泛化能力的研究者。

时序句子定位(TSG)旨在识别未剪辑视频中与给定文本查询语义对应的时间片段。尽管现有研究取得显著进展,但常忽略文本与视频之间的虚假相关性。这种虚假相关性主要源于两方面:(1) 文本数据固有的偏差,如特定动词或短语频繁共现;(2) 模型对视频内容中显著或重复模式的过拟合。这些偏差导致模型错误关联文本线索与视觉片段,造成预测不可靠且泛化能力差。为此,本文提出一种基于因果推断的新框架——因果干预与反事实推理,以消除虚假相关性并增强模型鲁棒性。具体而言,首先从因果视角建立结构因果模型;针对反映文本对特定动词/短语偏见的未观测混杂因子,引入文本因果干预,利用do-演算估计因果效应;进一步通过构建仅聚焦视频特征的反事实场景,排除查询与多模态融合特征的影响,实现对视频影响的剥离。实验在多个公开数据集上验证了该方法的优势,代码已开源。

原文摘要 · Abstract (English)

Temporal Sentence Grounding (TSG) aims to identify relevant moments in an untrimmed video that semantically correspond to a given textual query. Despite existing studies having made substantial progress, they often overlook the issue of spurious correlations between video and textual queries. These spurious correlations arise from two primary factors: (1) inherent biases in the textual data, such as frequent co-occurrences of specific verbs or phrases, and (2) the model's tendency to overfit to salient or repetitive patterns in video content. Such biases mislead the model into associating textual cues with incorrect visual moments, resulting in unreliable predictions and poor generalization to out-of-distribution examples. To overcome these limitations, we propose a novel TSG framework, causal intervention and counterfactual reasoning that utilizes causal inference to eliminate spurious correlations and enhance the model's robustness. Specifically, we first formulate the TSG task from a causal perspective with a structural causal model. Then, to address unobserved confounders reflecting textual biases toward specific verbs or phrases, a textual causal intervention is proposed, utilizing do-calculus to estimate the causal effects. Furthermore, visual counterfactual reasoning is performed by constructing a counterfactual scenario that focuses solely on video features, excluding the query and fused multi-modal features. This allows us to debias the model by isolating and removing the influence of the video from the overall effect. Experiments on public datasets demonstrate the superiority of the proposed method. The code is available at https://github.com/Tangkfan/CICR.

时序定位因果推理视频理解

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。