用文字描述和图像增强3D点云分割，提升细粒度识别能力。
All in One: Visual-Description-Guided Unified Point Cloud Segmentation
- 融合CLIP和大语言模型生成文本描述，引入多模态线索
- 在ScanNet、S3DIS上实现语义/实例/全景分割新纪录
- 适合需要高精度3D场景理解的机器人与自动驾驶应用
统一的3D点云分割对场景理解至关重要，但受限于稀疏结构、标注有限以及复杂环境中细粒度类别区分困难。现有方法因监督不足和多模态线索匮乏，难以捕捉丰富语义与上下文信息，导致类别与实例区分效果不佳。为此，我们提出VDG-Uni3DSeg框架，融合预训练视觉-语言模型（如CLIP）和大语言模型（LLMs），利用互联网获取的参考图像与LLM生成的文本描述，引入丰富的多模态线索，实现细粒度类别与实例分离。设计语义-视觉对比损失对齐点特征与多模态查询，并引入空间增强模块高效建模全局场景关系。在封闭集范式下，基于离线生成的多模态知识，该方法在语义、实例和全景分割任务上均达到当前最优性能，为3D理解提供可扩展、实用的解决方案。代码已开源：https://github.com/Hanzy1996/VDG-Uni3DSeg。
原文摘要 · Abstract (English)
Unified segmentation of 3D point clouds is crucial for scene understanding, but is hindered by its sparse structure, limited annotations, and the challenge of distinguishing fine-grained object classes in complex environments. Existing methods often struggle to capture rich semantic and contextual information due to limited supervision and a lack of diverse multimodal cues, leading to suboptimal differentiation of classes and instances. To address these challenges, we propose VDG-Uni3DSeg, a novel framework that integrates pre-trained vision-language models (e.g., CLIP) and large language models (LLMs) to enhance 3D segmentation. By leveraging LLM-generated textual descriptions and reference images from the internet, our method incorporates rich multimodal cues, facilitating fine-grained class and instance separation. We further design a Semantic-Visual Contrastive Loss to align point features with multimodal queries and a Spatial Enhanced Module to model scene-wide relationships efficiently. Operating within a closed-set paradigm that utilizes multimodal knowledge generated offline, VDG-Uni3DSeg achieves state-of-the-art results in semantic, instance, and panoptic segmentation, offering a scalable and practical solution for 3D understanding. Our code is available at https://github.com/Hanzy1996/VDG-Uni3DSeg.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。