将3D高斯点云的语义理解从嵌入式改为外在索引,提升开放词汇识别效率与精度。
ExtrinSplat: Decoupling Geometry and Semantics for Open-Vocabulary Understanding in 3D Gaussian Splatting
- 用视觉语言模型对高斯点聚类分组,生成轻量文本假设替代传统特征嵌入
- 场景适配时间从小时级降至分钟级,存储开销降低数个数量级
- 支持复杂语义多义性,适合需要高效开放词汇理解的3D场景应用
将2D开放词汇理解能力迁移至3D高斯喷溅(3DGS)场景是一项关键挑战。主流基于嵌入范式的方案存在三方面缺陷:(i) 几何-语义不一致,以点而非物体为语义基础,限制语义保真度;(ii) 语义膨胀,将数十GB特征数据注入几何结构;(iii) 语义僵化,每个高斯点仅支持单一特征难以捕捉丰富多义性。为此,我们提出ExtrinSplat,一种基于外在范式的框架,实现几何与语义解耦。该框架不嵌入特征,而是将高斯点聚类为多层次、重叠的3D物体组,再由视觉语言模型(VLM)解析这些组,生成轻量级文本假设,构建原生支持复杂多义性的外在索引层。通过以轻量索引替代昂贵特征嵌入,ExtrinSplat将场景适配时间从数小时缩短至分钟级,存储开销降低数个数量级。在开放词汇3D物体选择与语义分割基准任务中,ExtrinSplat显著优于现有嵌入式框架,验证了外在范式的有效性与高效性。
原文摘要 · Abstract (English)
Lifting 2D open-vocabulary understanding into 3D Gaussian Splatting (3DGS) scenes is a critical challenge. Mainstream methods, built on an embedding paradigm, suffer from three key flaws: (i) geometry-semantic inconsistency, where points, rather than objects, serve as the semantic basis, limiting semantic fidelity; (ii) semantic bloat from injecting gigabytes of feature data into the geometry; and (iii) semantic rigidity, as one feature per Gaussian struggles to capture rich polysemy. To overcome these limitations, we introduce ExtrinSplat, a framework built on the extrinsic paradigm that decouples geometry from semantics. Instead of embedding features, ExtrinSplat clusters Gaussians into multi-granularity, overlapping 3D object groups. A Vision-Language Model (VLM) then interprets these groups to generate lightweight textual hypotheses, creating an extrinsic index layer that natively supports complex polysemy. By replacing costly feature embedding with lightweight indices, ExtrinSplat reduces scene adaptation time from hours to minutes and lowers storage overhead by several orders of magnitude. On benchmark tasks for open-vocabulary 3D object selection and semantic segmentation, ExtrinSplat outperforms established embedding-based frameworks, validating the efficacy and efficiency of the proposed extrinsic paradigm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。