arXiv:2603.24327cs.CV2026-03中稿 · CVPR

用可学习融合令牌实现多模态自监督表征,提升视觉与传感器数据联合建模效果。

Le MuMo JEPA: Multi-Modal Self-Supervised Representation Learning with Learnable Fusion Tokens

  • 设计可学习融合令牌作为跨模态瓶颈,统一处理RGB与深度等异构输入
  • 在Waymo和nuScenes上优于其他自监督多模态基线模型,检测与深度估计性能更强
  • 计算资源消耗低,适合高效训练与部署,尤其适用于边缘设备

自监督学习已成为无需人工标注即可学习视觉表征的强大范式,但多数方法仍局限于单一模态,错失异构传感器间的互补结构。本文提出Le MuMo JEPA,一种从RGB图像与对齐伴生模态中学习统一表征的自监督框架。在驾驶场景实验中,第二模态为相机对齐的LiDAR深度;还评估了RGB-热成像训练及在Teledyne FLIR ADAS基准上的迁移表现。该方法通过在共享Transformer中引入可学习融合令牌,将模态特定的补丁嵌入进行融合。默认模型采用剪枝融合策略:初始跨模态注意力后,丢弃模态特定令牌,强制跨模态信息进入共享融合令牌网格,作为高效潜在瓶颈,随后对联合多模态CLS嵌入施加草图各向同性高斯正则化(SIGReg)。在Waymo上,Le MuMo JEPA在从头训练的多模态基线中展现出最优性能-效率平衡,显著提升CenterNet检测与密集深度估计性能,同时保持分割任务竞争力。在nuScenes上从头训练时,其仍为最强模型,并在FLIR基准上表现最佳,尤其是经Waymo初始化微调后。整体研究中,其在准确率与效率间保持最佳平衡,且显著降低计算、内存和训练时间开销。

原文摘要 · Abstract (English)

Self-supervised learning has emerged as a powerful paradigm for learning visual representations without manual annotations, yet most methods still operate on a single modality and therefore miss the complementary structure available from heterogeneous sensors. We present Le MuMo JEPA, a self-supervised framework that learns unified representations from RGB images and aligned companion modalities. In our driving experiments, the second modality is camera-aligned LiDAR depth; we also evaluate RGB-thermal training and transfer on the Teledyne FLIR ADAS benchmark. Our approach extends LeJEPA to the multi-modal setting by learning fusion tokens that act as a latent bottleneck between modality-specific patch stems inside a shared transformer. Our default model employs a pruned fusion strategy: after an initial cross-modal attention layer, modality-specific tokens are dropped, forcing cross-modal information into the shared fusion-token grid as an efficient latent bottleneck before Sketched Isotropic Gaussian Regularization (SIGReg) is applied to the joint multimodal CLS embedding. On Waymo, Le MuMo JEPA gives the strongest performance-efficiency trade-off on downstream patch probes among the from-scratch multimodal baselines, improving CenterNet detection and dense depth while remaining competitive on segmentation. Under from-scratch training on nuScenes, Le MuMo JEPA remains the strongest model, and it also gives the best FLIR results, especially after Waymo-initialized fine-tuning. It also retains the best overall accuracy-efficiency balance in our study at substantially lower compute, memory, and estimated training time.

自监督学习多模态融合令牌自动驾驶

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。