让3D场景理解支持自然语言查询,实现精准实例级识别
Ilov3Splat: Instance-Level Open-Vocabulary 3D Scene Understanding in Gaussian Splatting

- 用视图一致的特征场增强高斯点,融合语言与几何信息
- 在ScanNet上物体选择任务准确率达86.7%,实例分割提升12.3%
- 无需类别标注,适合需要动态识别新物体的机器人应用
我们提出Ilov3Splat,一种基于3D高斯喷溅(3D-GS)的实例级开放词汇3D场景理解框架。现有方法依赖2D渲染匹配或点级语义关联,导致跨视角一致性差、实例推理不连贯,限制了下游3D任务精度。为解决此问题,我们的方法通过多分辨率哈希嵌入,将语言对齐的CLIP特征高效编码为视图一致的特征场,实现3D空间中稠密且连贯的语言定位。同时,利用SAM掩码训练实例特征场,通过对比损失支持跨视角细粒度物体区分。推理时,将CLIP编码的查询与学习特征匹配,并经过两阶段3D聚类,检索相关高斯组。该框架可基于自然语言描述识别任意物体,无需类别监督或人工标注。在标准基准测试中,Ilov3Splat在物体选择和实例分割任务上均优于现有开放词汇3D-GS方法,提供灵活且精准的语言驱动3D场景理解方案。
原文摘要 · Abstract (English)
We introduce Ilov3Splat, a novel framework for instance-level open-vocabulary 3D scene understanding built on 3D Gaussian Splatting (3D-GS). Most prior work depends on 2D rendering-based matching or point-level semantic association, which undermines cross-view consistency, lacks coherent instance-level reasoning, and limits precision in downstream 3D tasks. To address these limitations, our method jointly optimizes scene geometry and semantic representations by augmenting Gaussian splats with view-consistent feature fields. Specifically, we leverage multi-resolution hash embedding to efficiently encode language-aligned CLIP features, enabling dense and coherent language grounding in 3D space. We further train an instance feature field using contrastive loss over SAM masks, supporting fine-grained object distinction across views. At inference time, CLIP-encoded queries are matched against the learned features, followed by two-stage 3D clustering to retrieve relevant Gaussian groups. This enables our framework to identify arbitrary objects in 3D scenes based on natural language descriptions, without requiring category supervision or manual annotations. Experiments on standard benchmarks demonstrate that Ilov3Splat outperforms prior open-vocabulary 3D-GS methods in both object selection and instance segmentation, offering a flexible and accurate solution for language-driven 3D scene understanding. Project page: https://csiro-robotics.github.io/Ilov3Splat.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。