提出多模态因果模型，精准理解自动驾驶视频中的驾驶行为因果关系。
MCAM: Multimodal Causal Analysis Model for Ego-Vehicle-Level Driving Video Understanding
- 构建视觉与语言模态间的潜在因果结构，动态建模驾驶状态图。
- 在BDD-X和CoVLA数据集上达到最新最优性能，显著提升因果推理能力。
- 适合自动驾驶场景下的行为理解与决策分析研究者使用。
准确的驾驶行为识别与推理对自动驾驶视频理解至关重要。现有方法常局限于浅层因果关系，无法处理跨模态伪相关性，且忽视了车辆自身层面的因果建模。为此，我们提出一种新型多模态因果分析模型（MCAM），在视觉与语言模态间构建潜在因果结构。首先，设计多层级特征提取器以捕捉长时依赖；其次，通过有向无环图（DAG）动态建模驾驶状态的因果关系；最后，利用视觉-语言变换器对齐关键视觉特征与其对应语义表达。在BDD-X和CoVLA数据集上的大量实验表明，MCAM在视觉-语言因果关系学习中达到当前最优表现。此外，模型在捕捉视频序列内的因果特性方面展现出优异能力，验证了其在自动驾驶应用中的有效性。代码已开源：https://github.com/SixCorePeach/MCAM。
原文摘要 · Abstract (English)
Accurate driving behavior recognition and reasoning are critical for autonomous driving video understanding. However, existing methods often tend to dig out the shallow causal, fail to address spurious correlations across modalities, and ignore the ego-vehicle level causality modeling. To overcome these limitations, we propose a novel Multimodal Causal Analysis Model (MCAM) that constructs latent causal structures between visual and language modalities. Firstly, we design a multi-level feature extractor to capture long-range dependencies. Secondly, we design a causal analysis module that dynamically models driving scenarios using a directed acyclic graph (DAG) of driving states. Thirdly, we utilize a vision-language transformer to align critical visual features with their corresponding linguistic expressions. Extensive experiments on the BDD-X, and CoVLA datasets demonstrate that MCAM achieves SOTA performance in visual-language causal relationship learning. Furthermore, the model exhibits superior capability in capturing causal characteristics within video sequences, showcasing its effectiveness for autonomous driving applications. The code is available at https://github.com/SixCorePeach/MCAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。