用语言模型教视觉模型做假设推理,提升视频理解能力。
Distilling Counterfactual Reasoning from Language to Vision: Causal Graph Guided Post-Training for Video Understanding
- 从语言模型中提取反事实推理能力,迁移给视觉模型
- 在复杂多步因果链任务上,性能显著优于基线模型
- 适合研究视频理解与因果推理的学者使用
视觉语言模型(VLMs)在视频理解方面取得显著进展,尤其在特征对齐、事件推理和指令遵循任务中表现突出。然而,其反事实推理能力——即在假设条件下推断替代结果的能力——仍待深入探索。该能力对鲁棒视频理解至关重要,需识别潜在因果结构并推理未观测可能性,而非仅识别已有模式。为此,我们提出CounterVQA,一个基于视频的基准测试,包含三个渐进难度层级,评估反事实推理的不同维度。对主流开源与闭源模型的全面评估显示,尽管模型在简单反事实问题上表现良好,但在复杂多步因果链任务上性能急剧下降。为解决此问题,我们提出一种后训练方法CFGPT,通过从语言模态中蒸馏反事实推理能力,有效提升视觉模型的反事实推理能力,在所有CounterVQA难度层级上均实现稳定提升。数据集与代码将后续发布。
原文摘要 · Abstract (English)
Vision Language Models (VLMs) have recently shown significant advancements in video understanding, especially in feature alignment, event reasoning, and instruction-following tasks. However, their capability for counterfactual reasoning, inferring alternative outcomes under hypothetical conditions, remains underexplored. This capability is essential for robust video understanding, as it requires identifying underlying causal structures and reasoning about unobserved possibilities, rather than merely recognizing observed patterns. To systematically evaluate this capability, we introduce CounterVQA, a video-based benchmark featuring three progressive difficulty levels that assess different aspects of counterfactual reasoning. Through comprehensive evaluation of both state-of-the-art open-source and closed-source models, we uncover a substantial performance gap: while these models achieve reasonable accuracy on simple counterfactual questions, performance degrades significantly on complex multi-hop causal chains. To address these limitations, we develop a post-training method, CFGPT, that enhances a model's visual counterfactual reasoning ability by distilling its counterfactual reasoning capability from the language modality, yielding consistent improvements across all CounterVQA difficulty levels. Dataset and code will be further released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。