用视觉大模型提升点云语义分割在不同激光雷达间的泛化能力
Improving Multimodal Distillation for 3D Semantic Segmentation under Domain Shift
- 基于图像到点云的知识蒸馏，优化点云主干网络结构以增强跨域适应性
- 冻结预训练主干+轻量MLP头可实现最佳性能，支持一次性预训练应对多种域偏移
- 在四个挑战性场景中达到当前最优，适合跨设备点云分割研究者参考
在特定激光雷达上全监督训练的语义分割网络难以泛化到未见激光雷达。为缩小域偏移带来的性能差距，近期趋势是利用视觉基础模型（VFMs）提供跨域鲁棒特征。本文系统研究了如何在无监督域自适应场景中有效利用VFMs进行点云语义分割。基于无监督图像到点云知识蒸馏，研究发现：(1) 点云主干网络架构对目标域泛化性能至关重要；(2) 可一次性预训练单个主干模型，用于应对多种域偏移；(3) 最优结果来自冻结预训练主干并仅训练一个MLP头部进行分割。该流程在四个广泛认可且具有挑战性的设置中达到当前最优。代码将公开于：https://github.com/valeoai/muddos。
原文摘要 · Abstract (English)
Semantic segmentation networks trained under full supervision for one type of lidar fail to generalize to unseen lidars without intervention. To reduce the performance gap under domain shifts, a recent trend is to leverage vision foundation models (VFMs) providing robust features across domains. In this work, we conduct an exhaustive study to identify recipes for exploiting VFMs in unsupervised domain adaptation for semantic segmentation of lidar point clouds. Building upon unsupervised image-to-lidar knowledge distillation, our study reveals that: (1) the architecture of the lidar backbone is key to maximize the generalization performance on a target domain; (2) it is possible to pretrain a single backbone once and for all, and use it to address many domain shifts; (3) best results are obtained by keeping the pretrained backbone frozen and training an MLP head for semantic segmentation. The resulting pipeline achieves state-of-the-art results in four widely-recognized and challenging settings. The code will be available at: https://github.com/valeoai/muddos.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。