将2D语义特征高效融入3D高斯点云，实现任意视角的开放词汇理解。
GAGS: Granularity-Aware Feature Distillation for Language Gaussian Splatting
- 通过提示点密度与相机距离关联提升多视角分割一致性
- 引入无监督学习的粒度因子，筛选稳定一致的2D特征
- 推理速度比基线快2倍，适用于开放词汇场景理解
三维开放词汇场景理解近年来受到广泛关注，旨在精确感知空间中物体的复杂语义属性。本文提出GAGS框架，将2D CLIP特征蒸馏至3D高斯点云，支持任意视角下的开放词汇查询。其主要挑战在于2D特征提取存在多视角不一致性，导致3D特征场监督不稳定。GAGS采用两项新策略：首先，将SAM的提示点密度与相机距离关联，显著提升多视角分割一致性；其次，解码粒度因子以引导蒸馏过程，该因子可无监督学习，仅选择多视角一致的2D特征。在两个数据集上的实验表明，GAGS在视觉定位与语义分割任务中均取得显著性能与稳定性提升，推理速度比基线方法快2倍。代码与附加结果见https://pz0826.github.io/GAGS-Webpage/。
原文摘要 · Abstract (English)
3D open-vocabulary scene understanding, which accurately perceives complex semantic properties of objects in space, has gained significant attention in recent years. In this paper, we propose GAGS, a framework that distills 2D CLIP features into 3D Gaussian splatting, enabling open-vocabulary queries for renderings on arbitrary viewpoints. The main challenge of distilling 2D features for 3D fields lies in the multiview inconsistency of extracted 2D features, which provides unstable supervision for the 3D feature field. GAGS addresses this challenge with two novel strategies. First, GAGS associates the prompt point density of SAM with the camera distances, which significantly improves the multiview consistency of segmentation results. Second, GAGS further decodes a granularity factor to guide the distillation process and this granularity factor can be learned in a unsupervised manner to only select the multiview consistent 2D features in the distillation process. Experimental results on two datasets demonstrate significant performance and stability improvements of GAGS in visual grounding and semantic segmentation, with an inference speed 2$\times$ faster than baseline methods. The code and additional results are available at https://pz0826.github.io/GAGS-Webpage/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。