arXiv:2410.19446cs.CV2024-10被引 11

通过融合后蒸馏,实现跨模态正样本对齐,提升3D语义分割的域适应性能。

Fusion-then-Distillation: Toward Cross-modal Positive Distillation for Domain Adaptive 3D Semantic Segmentation

  • 先融合多模态特征,再进行跨域正样本蒸馏,增强模态间互补性。
  • 在多个无监督和半监督场景下达到当前最优效果,显著提升分割精度。
  • 适合需要跨模态域适应的3D视觉任务,如自动驾驶点云分析。

在跨模态无监督域自适应中,模型需在无目标域标注的情况下,将源域(如合成数据)训练的模型适配至目标域(如真实世界)。以往方法通过相互模仿不同域的跨模态输出,强制类概率分布一致,但忽略了跨模态融合带来的互补性。为此,本文提出新型融合-蒸馏方法(FtD++),探索源域与目标域间的跨模态正样本蒸馏。FtD++包含三个关键组件:首先,设计一个模型无关的特征融合模块,生成跨模态融合表示,建立潜在空间并强制两模态最大相关性和互补性;其次,提出跨模态正蒸馏机制,保留多模态输入的完整信息,结合源域语义与目标域风格,实现域-模态对齐;最后,引入跨模态去偏伪标签策略,通过自训练建模伪标签不确定性。大量实验表明,FtD++在多种无监督与半监督域自适应场景下均达到领先性能。代码已开源:https://github.com/Barcaaaa/FtD-PlusPlus。

原文摘要 · Abstract (English)

In cross-modal unsupervised domain adaptation, a model trained on source-domain data (e.g., synthetic) is adapted to target-domain data (e.g., real-world) without access to target annotation. Previous methods seek to mutually mimic cross-modal outputs in each domain, which enforces a class probability distribution that is agreeable in different domains. However, they overlook the complementarity brought by the heterogeneous fusion in cross-modal learning. In light of this, we propose a novel fusion-then-distillation (FtD++) method to explore cross-modal positive distillation of the source and target domains for 3D semantic segmentation. FtD++ realizes distribution consistency between outputs not only for 2D images and 3D point clouds but also for source-domain and augment-domain. Specially, our method contains three key ingredients. First, we present a model-agnostic feature fusion module to generate the cross-modal fusion representation for establishing a latent space. In this space, two modalities are enforced maximum correlation and complementarity. Second, the proposed cross-modal positive distillation preserves the complete information of multi-modal input and combines the semantic content of the source domain with the style of the target domain, thereby achieving domain-modality alignment. Finally, cross-modal debiased pseudo-labeling is devised to model the uncertainty of pseudo-labels via a self-training manner. Extensive experiments report state-of-the-art results on several domain adaptive scenarios under unsupervised and semi-supervised settings. Code is available at https://github.com/Barcaaaa/FtD-PlusPlus.

3D分割域自适应跨模态蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。