将2D视觉特征无训练提升至3D高斯点云场景，速度快且效果佳。
LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenes
- 用图扩散机制聚合2D特征，无需训练直接升级到3D
- DINOv2特征在分割任务上媲美专用模型，速度更快
- 适配CLIP等多类特征，适用于开放词汇分割场景
我们解决视觉基础模型（如DINO、SAM、CLIP）拓展至3D任务的难题。提出一种新方法，将2D图像特征无训练地提升为3D高斯点云表示。不同于依赖重建损失的传统方法，本方法采用更简单高效的特征聚合技术，并结合图扩散机制，利用DINOv2诱导的成对相似性和3D几何结构精炼3D特征（如粗略分割掩码）。在多个下游任务中达到与当前最优方法相当的性能，同时实现显著提速。特别地，仅使用通用的DINOv2特征即获得具有竞争力的分割结果，尽管DINOv2未在数百万标注分割掩码上训练；应用于CLIP特征时，在开放词汇物体分割任务中表现优异，体现方法的通用性。
原文摘要 · Abstract (English)
We address the problem of extending the capabilities of vision foundation models such as DINO, SAM, and CLIP, to 3D tasks. Specifically, we introduce a novel method to uplift 2D image features into Gaussian Splatting representations of 3D scenes. Unlike traditional approaches that rely on minimizing a reconstruction loss, our method employs a simpler and more efficient feature aggregation technique, augmented by a graph diffusion mechanism. Graph diffusion refines 3D features, such as coarse segmentation masks, by leveraging 3D geometry and pairwise similarities induced by DINOv2. Our approach achieves performance comparable to the state of the art on multiple downstream tasks while delivering significant speed-ups. Notably, we obtain competitive segmentation results using only generic DINOv2 features, despite DINOv2 not being trained on millions of annotated segmentation masks like SAM. When applied to CLIP features, our method demonstrates strong performance in open-vocabulary object segmentation tasks, highlighting the versatility of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。