用多尺度法向量分布将3D场景转为语义令牌,实现统一的视觉语言理解。
Scenes as Tokens: Multi-Scale Normal Distributions Transform Tokenizer for General 3D Vision-Language Understanding
- 基于多尺度法向量分布构建3D场景表示,分阶段生成统一语义令牌
- 在3D指代分割、视觉问答等任务上显著提升性能,最高达12.6%精度
- 支持点、框、掩码等交互式提示,适合多任务通用3D理解场景
近期3D视觉语言模型(VLMs)展现出强大的3D场景理解与推理潜力。然而,如何将3D场景有效转化为整体语义令牌,并在多种任务中复用这些令牌,仍是重大挑战。我们提出NDTokenizer3D,一种通用型3D VLM,可完成多样化的3D场景理解任务,并自然支持人机交互,实现语言推理与3D空间理解的融合。核心方法是一个三阶段场景令牌化流程,基于多尺度法向量分布变换(NDT)表示,搭配多尺度NDT解码器(MSDec)。首先,从高分辨率点云构建多尺度NDT表示,同时保留全局上下文与细粒度几何细节;其次,通过MSDec逐步融合跨尺度NDT特征,生成可被大语言模型直接使用的整体场景令牌。此外,MSDec还可作为通用接口,支持点、框、掩码等交互式提示及分割掩码解码,将多样化3D理解任务统一于同一架构中。凭借紧凑而统一的设计,NDTokenizer3D实现了在3D指代分割、3D视觉问答和3D密集描述任务上的显著提升。
原文摘要 · Abstract (English)
Recent advances in 3D vision-language models (VLMs) highlight a strong potential for 3D scene understanding and reasoning. However, effectively tokenizing 3D scenes into holistic scene tokens, and leveraging these tokens across diverse 3D understanding tasks, remain highly challenging. We present NDTokenizer3D, a generalist 3D VLM that performs a wide range of 3D scene understanding tasks while naturally supporting human interactions, thereby bridging language-level reasoning with 3D spatial understanding. The core of our approach is a novel three-stage scene tokenization pipeline built upon a Multi-Scale Normal Distributions Transform (NDT) representation, paired with a Multi-Scale NDT Decoder (MSDec). Specifically, NDTokenizer3D first constructs a multi-scale NDT representation from raw high-resolution point clouds, preserving both global context and fine-grained geometric details. Next, the MSDec progressively fuses cross-scale NDT features, producing holistic scene tokens consumable by LLM endpoints. Beyond tokenization, MSDec is repurposed as a general interface for human-interactive prompting (points, boxes, masks) and segmentation-mask decoding, unifying diverse 3D scene understanding tasks within a single architecture. With this compact and unified design, NDTokenizer3D offers a fine-grained, general-purpose 3D VLM, achieving remarkable improvements in 3D Referring Segmentation, 3D Visual Question Answering, and 3D Dense Captioning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。