不用3D标注和训练,用语言空间实现精准3D场景理解。
GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting

- 用视觉语言模型做翻译,生成图像的结构化文本描述
- 在纯语言空间中聚合信息,实现零3D训练与标注的语义分割
- 能定位未见物体,且结果可解释,适合文化遗产等场景
开放词汇3D语义分割方法通常将CLIP特征映射到3D空间,但这种联合视觉-语言空间在组合任务中表现如词袋模型。现有无标注方法仍需大规模3D训练数据集及领域专用3D编码器。本文提出全新范式:仅将视觉语言模型作为翻译器,为每张姿态图像生成结构化的实体级描述。这些描述被对齐、投影并直接聚合至通用的纯语言嵌入空间,无需任何3D训练数据或编码器。在ScanNet++上,性能媲美基于ScanNet训练的强基准;在5栋建筑的文化遗产数据集上,原始分数偏好基于CLIP的变体,但一次系统性词汇修正即逆转排名,另一独立修正进一步验证该效应,表明语言空间嵌入更忠实反映物理内容。该保真度延伸至ScanNet++上的真正未见物体(OOV),证明语言空间嵌入对存在与不存在物体的区分远优于CLIP。GoDeep还能在无2D-3D标注条件下定位这些未见物体,所有表示均为离散文本,支持点级可解释预测。此外,结合启发式加权策略(优先精确而非频繁观测)与可解释性,我们提出一种聚合策略,作为概念验证,可实现更精细的局部定位。
原文摘要 · Abstract (English)
Open vocabulary 3D semantic segmentation methods typically lift CLIP features into 3D. This embeds points in a joint vision-language space known to behave like a bag-of-words on compositional tasks. Furthermore, even annotation free variants often require a large 3D training corpus and a dedicated 3D encoder per domain. Instead we use a vision-language model purely as a translator. It produces structured, entity-level descriptions of each posed image. These descriptions are grounded, projected, and aggregated directly in a general-purpose, language-only embedding space, with no 3D training corpus or encoder required. On ScanNet++, our pipeline is competitive with strong annotation free baselines trained on ScanNet. On a 5-building cultural heritage benchmark, raw scores initially favor a CLIP-based variant, but a single systematic vocabulary correction reverses this ranking. An effect confirmed by a second, independent correction on a different class, indicating that language-space embeddings track physical content more faithfully. This fidelity extends to genuinely out-of-vocabulary (OOV) objects on ScanNet++ proving that language-space embeddings separate presence from absence objects far more sharply than CLIP-based embeddings do. GoDeep also localize these OOV objects within the scene, all without any 2D-3D annotation. Because every representation remains discrete text, predictions are also explainable at the point level. Finally, exploiting both a heuristic weighting, that favors precise over merely frequent observations and GoDeep's explainability property, we propose an aggregation strategy, as a proof of concept, that favors finer elements localization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。