arXiv:2606.02352cs.CV2026-06中稿 · the IEEE ITSC 2026

提出多模态视频表示对齐新方法,提升驾驶分心检测的鲁棒性。

Multi-modal Video Representation Alignment for Robust Self-supervised Driver Distraction Detection

论文配图:Multi-modal Video Representation Alignment for Robust Self-supervised Driver Distraction Detection
图 1 · 摘自论文原文
  • 用循环一致性生成软目标,缓解负样本偏差问题。
  • 在Drive&Act数据集上跨模态性能超越基线,尤其在未知视角下表现更优。
  • 适合需要多传感器融合的自动驾驶场景,可扩展至复杂真实环境。

在真实场景中,如驾驶分心检测,多模态视频表征的鲁棒自监督学习至关重要,多个传感器提供互补但含噪信号。传统对比学习(如InfoNCE)假设所有负样本同等有效且正样本可靠,但在多模态数据中常因视角变化、遮挡或语义重叠而失效。本文提出一种新型多模态全局对齐框架,联合建模错误负样本与不可靠/错误正样本。引入基于循环一致性得分的软目标以放松硬负样本假设,并设计基于相似度分布的加权机制以减轻噪声正样本影响。方法将传统成对对齐拓展为有原则的全局多模态设置,聚合所有模态对的对齐信息。在Drive&Act数据集上的评估表明,该方法在RGB、IR、Depth和Skeleton模态上均持续优于成对及现有全局对齐基线。跨视角消融实验进一步验证其对未见相机视角的强大泛化能力,凸显表示的鲁棒性。整体框架为自监督全局多模态表示学习提供可扩展、高效解决方案,支持可靠的驾驶分心检测,并推动真实世界多模态视频理解发展。代码将开源于GitHub。

原文摘要 · Abstract (English)

Robust self-supervised learning of multi-modal video representations is critical for real-world applications such as driver distraction detection, where multiple sensors provide complementary but noisy signals. Conventional contrastive objectives, such as InfoNCE, assume all negatives are equally informative and all positives are reliable. However, this assumption is frequently violated in multi-modal data due to viewpoint changes, occlusions, or semantic overlap across modalities. In this work, we propose a novel framework for multi-modal global alignment that addresses these challenges by jointly modeling faulty negatives and unreliable or faulty positives. We introduce soft targets derived from cycle-consistency scores to relax the hard-negative assumption, and a weighting mechanism based on similarity distributions to mitigate the impact of noisy or faulty positives. Our approach extends traditional pairwise alignment to a principled global multi-modal setting, aggregating alignment information across all modality pairs. We evaluate our method on the Drive&Act dataset, demonstrating that it consistently outperforms both pairwise and existing global alignment baselines across RGB, IR, Depth, and Skeleton modalities. Cross-view ablation studies further show strong generalization to unseen camera perspectives, highlighting the robustness of our representations. Overall, our framework provides a scalable and effective solution for self-supervised global multi-modal representation learning, enabling reliable driver distraction detection and pioneering in real-world multi-modal video understanding. Our code will be published on GitHub.

多模态自监督驾驶检测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。