让3D场景能用对话交互,支持物体、视角和整体场景的聊天互动。
ChatSplat: 3D Conversational Gaussian Splatting
- 用视图级令牌与大模型结合,实现多视角对话理解。
- 提出分离掩码与语言特征的新结构,提升物体级交互灵活性。
- 引入可学习归一化,解决语言嵌入复杂分布带来的训练难题。
人类自然地通过语言与三维环境交互,构建用于场景理解与交互的3D语言场受到越来越多关注。本文提出ChatSplat系统,构建3D语言场,实现三维空间内丰富的基于聊天的交互。不同于以往主要使用聚焦于分割的CLIP衍生语言特征的方法,ChatSplat支持三个层级的交互:物体、视角和整个3D场景。在视图级交互中,设计编码器将每个视角的渲染特征图编码为令牌,再交由大语言模型(LLM)处理对话;在场景级交互中,融合多视角令牌,实现对全局场景的上下文理解;在物体级交互中,采用逐块语言嵌入,区别于LangSplat的像素级嵌入方式,显式分离掩码与特征表示,从而提升物体级交互的灵活性。针对大语言模型中语言嵌入分布复杂多变导致3D高斯学习困难的问题,提出可学习归一化技术,统一嵌入分布,促进有效训练。大量实验表明,ChatSplat可在三维空间中支持物体、视角和场景三个层级的交互,显著增强对场景的理解与用户参与度。
原文摘要 · Abstract (English)
Humans naturally interact with their 3D surroundings using language, and modeling 3D language fields for scene understanding and interaction has gained growing interest. This paper introduces ChatSplat, a system that constructs a 3D language field, enabling rich chat-based interaction within 3D space. Unlike existing methods that primarily use CLIP-derived language features focused solely on segmentation, ChatSplat facilitates interaction on three levels: objects, views, and the entire 3D scene. For view-level interaction, we designed an encoder that encodes the rendered feature map of each view into tokens, which are then processed by a large language model (LLM) for conversation. At the scene level, ChatSplat combines multi-view tokens, enabling interactions that consider the entire scene. For object-level interaction, ChatSplat uses a patch-wise language embedding, unlike LangSplat's pixel-wise language embedding that implicitly includes mask and embedding. Here, we explicitly decouple the language embedding into separate mask and feature map representations, allowing more flexible object-level interaction. To address the challenge of learning 3D Gaussians posed by the complex and diverse distribution of language embeddings used in the LLM, we introduce a learnable normalization technique to standardize these embeddings, facilitating effective learning. Extensive experimental results demonstrate that ChatSplat supports multi-level interactions -- object, view, and scene -- within 3D space, enhancing both understanding and engagement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。