arXiv:2505.24361cs.CV2025-05被引 5

提出新方法让单模态模型学会融合多模态信息,提升鲁棒性。

Revisiting Cross-Modal Knowledge Distillation: A Disentanglement Approach for RGBD Semantic Segmentation

  • 通过解耦表示与对比学习,分离训练RGB和深度图模型
  • 在三个数据集上优于现有方法,显著提升单模态推理性能
  • 适合传感器受限的机器人、自动驾驶等实际场景

RGB与深度(RGBD)多模态数据在机器人、自动驾驶和遥感等领域广泛应用。融合多模态数据可提供三维空间上下文,弥补标准RGB图像的不足。然而,推理时可能因传感器故障或资源限制无法获取所有模态,造成训练与推理模态不匹配。传统跨模态知识蒸馏(CMKD)采用教师-学生架构,但受限于教师模型选择与蒸馏过程设计,难以落地。为此,我们提出CroDiNo-KD:一种基于解耦表示的新跨模态知识蒸馏框架,通过对比学习与解耦数据增强,协同训练独立的单模态RGB与深度模型,以构建神经网络内部流形结构。我们在三个不同领域的RGBD数据集上评估该方法,并与近期CMKD框架对比。结果表明,CroDiNo-KD性能优异,提示应重新审视传统教师-学生范式在多模态到单模态知识迁移中的适用性。

原文摘要 · Abstract (English)

Multi-modal RGB and Depth (RGBD) data are predominant in many domains such as robotics, autonomous driving and remote sensing. The combination of these multi-modal data enhances environmental perception by providing 3D spatial context, which is absent in standard RGB images. Although RGBD multi-modal data can be available to train computer vision models, accessing all sensor modalities during the inference stage may be infeasible due to sensor failures or resource constraints, leading to a mismatch between data modalities available during training and inference. Traditional Cross-Modal Knowledge Distillation (CMKD) frameworks, developed to address this task, are typically based on a teacher/student paradigm, where a multi-modal teacher distills knowledge into a single-modality student model. However, these approaches face challenges in teacher architecture choices and distillation process selection, thus limiting their adoption in real-world scenarios. To overcome these issues, we introduce CroDiNo-KD (Cross-Modal Disentanglement: a New Outlook on Knowledge Distillation), a novel cross-modal knowledge distillation framework for RGBD semantic segmentation. Our approach simultaneously learns single-modality RGB and Depth models by exploiting disentanglement representation, contrastive learning and decoupled data augmentation with the aim to structure the internal manifolds of neural network models through interaction and collaboration. We evaluated CroDiNo-KD on three RGBD datasets across diverse domains, considering recent CMKD frameworks as competitors. Our findings illustrate the quality of CroDiNo-KD, and they suggest reconsidering the conventional teacher/student paradigm to distill information from multi-modal data to single-modality neural networks.

知识蒸馏多模态语义分割解耦表示

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。