通过知识蒸馏让3D模型快速实现开放词汇实例分割,速度提升6到152倍。
FOLK: Fast Open-Vocabulary 3D Instance Segmentation via Label-guided Knowledge Distillation
- 用2D视觉语言模型生成高质量标签一致的嵌入作为教师信号
- 3D学生模型直接从点云分类,实现35.7%的AP50性能
- 避免2D遮挡噪声,推理速度最快快152倍,适合实时应用
开放词汇3D实例分割旨在对未标注类别进行实例分割与分类。现有方法通常将3D实例映射到2D RGB-D图像,再使用视觉语言模型(VLMs)进行分类,但该策略易受2D遮挡影响,并带来高计算与内存开销,导致推理缓慢。为此,本文提出基于标签引导知识蒸馏的快速开放词汇3D实例分割方法(FOLK)。核心思想是设计一个教师模型,提取高质量的实例嵌入,并将其开放词汇知识蒸馏至3D学生模型。训练时,利用标签一致的2D CLIP嵌入指导3D学生模型学习,使其在推理阶段可直接从3D点云完成分类,避免2D遮挡引入的噪声,显著加速推理过程。在ScanNet200和Replica数据集上实验表明,FOLK在ScanNet200上达到35.7%的AP50,且推理速度比先前方法快6.0至152.2倍。代码将在论文接收后公开。
原文摘要 · Abstract (English)
Open-vocabulary 3D instance segmentation seeks to segment and classify instances beyond the annotated label space. Existing methods typically map 3D instances to 2D RGB-D images, and then employ vision-language models (VLMs) for classification. However, such a mapping strategy usually introduces noise from 2D occlusions and incurs substantial computational and memory costs during inference, slowing down the inference speed. To address the above problems, we propose a Fast Open-vocabulary 3D instance segmentation method via Label-guided Knowledge distillation (FOLK). Our core idea is to design a teacher model that extracts high-quality instance embeddings and distills its open-vocabulary knowledge into a 3D student model. In this way, during inference, the distilled 3D model can directly classify instances from the 3D point cloud, avoiding noise caused by occlusions and significantly accelerating the inference process. Specifically, we first design a teacher model to generate a 2D CLIP embedding for each 3D instance, incorporating both visibility and viewpoint diversity, which serves as the learning target for distillation. We then develop a 3D student model that directly produces a 3D embedding for each 3D instance. During training, we propose a label-guided distillation algorithm to distill open-vocabulary knowledge from label-consistent 2D embeddings into the student model. FOLK conducted experiments on the ScanNet200 and Replica datasets, achieving state-of-the-art performance on the ScanNet200 dataset with an AP50 score of 35.7, while running approximately 6.0x to 152.2x faster than previous methods. All codes will be released after the paper is accepted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。