让AI实时理解语言并映射到3D场景,无需预处理。
Online Language Splatting
- 每帧18毫秒生成高分辨率语言特征图
- 768维语言特征压缩至15维仍支持开放词汇
- 适合需要动态交互的机器人或AR应用
为使AI智能体能无缝与人类和三维环境交互,必须准确感知三维世界,并将人类语言与三维空间表示对齐。尽管已有研究通过将语言特征整合到基于3D高斯泼溅(3DGS)的几何精细场景表示中取得进展,但这些方法依赖于针对每张输入图像进行计算密集型的语言特征离线预处理,限制了在新环境中的适应性。本文提出首个在线、近实时、开放词汇的语言映射框架——Online Language Splatting,可直接集成于3DGS-SLAM系统中,无需预生成语言特征。核心挑战在于高效融合高维语言特征,同时兼顾计算速度、内存使用、渲染质量与开放词汇能力。为此,我们创新设计:(1) 每帧仅需18毫秒的高分辨率CLIP嵌入模块,生成细致语言特征图;(2) 两级在线自编码器,将768维CLIP特征压缩至15维,保持开放词汇能力;(3) 颜色-语言解耦优化策略,提升渲染质量。实验表明,该在线方法不仅在精度上超越最先进离线方法,效率提升超40倍,展现出在动态交互式AI应用中的巨大潜力。
原文摘要 · Abstract (English)
To enable AI agents to interact seamlessly with both humans and 3D environments, they must not only perceive the 3D world accurately but also align human language with 3D spatial representations. While prior work has made significant progress by integrating language features into geometrically detailed 3D scene representations using 3D Gaussian Splatting (GS), these approaches rely on computationally intensive offline preprocessing of language features for each input image, limiting adaptability to new environments. In this work, we introduce Online Language Splatting, the first framework to achieve online, near real-time, open-vocabulary language mapping within a 3DGS-SLAM system without requiring pre-generated language features. The key challenge lies in efficiently fusing high-dimensional language features into 3D representations while balancing the computation speed, memory usage, rendering quality and open-vocabulary capability. To this end, we innovatively design: (1) a high-resolution CLIP embedding module capable of generating detailed language feature maps in 18ms per frame, (2) a two-stage online auto-encoder that compresses 768-dimensional CLIP features to 15 dimensions while preserving open-vocabulary capabilities, and (3) a color-language disentangled optimization approach to improve rendering quality. Experimental results show that our online method not only surpasses the state-of-the-art offline methods in accuracy but also achieves more than 40x efficiency boost, demonstrating the potential for dynamic and interactive AI applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。