用2D大模型提升3D点云分割，效果更优且无需图像输入
DINO in the Room: Leveraging 2D Foundation Models for 3D Segmentation
- 将2D大模型特征投影到3D空间并注入分割网络
- 在室内外数据集上均达到当前最优性能
- 可离线蒸馏2D知识，推理时无需图像
视觉基础模型（VFMs）在大规模图像数据集上训练，提供了高质量特征，显著推动了2D视觉识别的发展。然而，尽管2D图像与3D点云数据集常同时存在，其在3D场景分割中的潜力仍被严重低估。虽然已有大量研究致力于2D-3D融合，但当前最先进的3D方法主要依赖3D数据，对整合VFMs的探索不足。本文提出DITR，一种通用方法：提取2D基础模型特征，将其投影至3D空间，并注入3D点云分割模型。DITR在室内与室外3D语义分割基准上均取得最先进结果。为支持推理阶段无图像输入的情况，我们进一步提出通过蒸馏2D基础模型来预训练3D模型。利用从2D VFMs中蒸馏的知识初始化3D主干网络，为下游3D分割任务奠定强基底，显著提升多个数据集上的性能。
原文摘要 · Abstract (English)
Vision foundation models (VFMs) trained on large-scale image datasets provide high-quality features that have significantly advanced 2D visual recognition. However, their potential in 3D scene segmentation remains largely untapped, despite the common availability of 2D images alongside 3D point cloud datasets. While significant research has been dedicated to 2D-3D fusion, recent state-of-the-art 3D methods predominantly focus on 3D data, leaving the integration of VFMs into 3D models underexplored. In this work, we challenge this trend by introducing DITR, a generally applicable approach that extracts 2D foundation model features, projects them to 3D, and finally injects them into a 3D point cloud segmentation model. DITR achieves state-of-the-art results on both indoor and outdoor 3D semantic segmentation benchmarks. To enable the use of VFMs even when images are unavailable during inference, we additionally propose to pretrain 3D models by distilling 2D foundation models. By initializing the 3D backbone with knowledge distilled from 2D VFMs, we create a strong basis for downstream 3D segmentation tasks, ultimately boosting performance across various datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。