让3D场景语义定位更快更准,支持模糊语言查询
FAST-Splat: Fast, Ambiguity-Free Semantics Transfer in Gaussian Splatting
- 直接给每个高斯点加语义码,不依赖额外神经网络
- 训练快6-8倍,渲染快18-51倍,显存减少约6倍
- 能准确回应模糊自然语言查询,适合交互式3D应用
我们提出FAST-Splat,一种快速、无歧义的语义高斯点云渲染方法,旨在解决现有语义高斯点云方法在训练与渲染速度慢、内存占用高、语义定位模糊等方面的局限。通过自底向上的设计,打破封闭词汇语义蒸馏的限制,实现开放词汇语义蒸馏。该方法直接为每个高斯点添加特定语义码,利用高斯点云表示的显式结构,保留了高斯点云在训练和渲染速度及内存使用上的优势。结合哈希表,可对开放词汇用户查询进行语义相似度匹配,输出明确的语义标签和3D掩码。实验表明,相比最优竞品,FAST-Splat训练速度提升6-8倍,渲染速度提升18-51倍,显存占用减少约6倍,同时保持相当或更优的语义分割性能。
原文摘要 · Abstract (English)
We present FAST-Splat for fast, ambiguity-free semantic Gaussian Splatting, which seeks to address the main limitations of existing semantic Gaussian Splatting methods, namely: slow training and rendering speeds; high memory usage; and ambiguous semantic object localization. We take a bottom-up approach in deriving FAST-Splat, dismantling the limitations of closed-set semantic distillation to enable open-set (open-vocabulary) semantic distillation. Ultimately, this key approach enables FAST-Splat to provide precise semantic object localization results, even when prompted with ambiguous user-provided natural-language queries. Further, by exploiting the explicit form of the Gaussian Splatting scene representation to the fullest extent, FAST-Splat retains the remarkable training and rendering speeds of Gaussian Splatting. Precisely, while existing semantic Gaussian Splatting methods distill semantics into a separate neural field or utilize neural models for dimensionality reduction, FAST-Splat directly augments each Gaussian with specific semantic codes, preserving the training, rendering, and memory-usage advantages of Gaussian Splatting over neural field methods. These Gaussian-specific semantic codes, together with a hash-table, enable semantic similarity to be measured with open-vocabulary user prompts and further enable FAST-Splat to respond with unambiguous semantic object labels and $3$D masks, unlike prior methods. In experiments, we demonstrate that FAST-Splat is 6x to 8x faster to train, achieves between 18x to 51x faster rendering speeds, and requires about 6x smaller GPU memory, compared to the best-competing semantic Gaussian Splatting methods. Further, FAST-Splat achieves relatively similar or better semantic segmentation performance compared to existing methods. After the review period, we will provide links to the project website and the codebase.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。