用2D大模型能力实现3D场景的精准语义分割,无需3D标注数据。
MLLM-For3D: Adapting Multimodal Large Language Model for 3D Reasoning Segmentation
- 将2D多模态大模型生成的伪分割图和文本嵌入投影到3D空间
- 在无3D标签情况下,性能超越现有方法,准确理解用户意图
- 通过空间一致性与词查询对齐,避免错误生成和对象不一致
推理分割旨在基于人类意图和空间推理,在复杂场景中分割目标物体。尽管近期多模态大语言模型(MLLMs)在2D图像推理分割上表现优异,但将其能力迁移到3D场景仍研究不足。本文提出MLLM-For3D,一种简单有效的框架,将2D MLLMs的知识迁移至3D场景理解。具体地,利用MLLM生成多视角伪分割掩码及对应文本嵌入,再将2D掩码反投影至3D空间并与文本嵌入对齐。主要挑战在于缺乏3D上下文和多视角间的空间一致性,导致模型幻觉生成不存在物体且目标识别不一致。使用此类无关物体训练3D模型会降低性能。为此,我们引入空间一致性策略,确保分割掩码在3D空间中保持连贯,有效捕捉场景几何结构。此外,设计了Token-for-Query方法实现跨视角的多模态语义对齐,保证同一物体在不同视角下被一致识别。在多个具有挑战性的室内场景基准上的大量实验表明,即使没有3D标注训练数据,MLLM-For3D也优于现有3D推理分割方法,能有效解析用户意图、理解3D场景并推理空间关系。
原文摘要 · Abstract (English)
Reasoning segmentation aims to segment target objects in complex scenes based on human intent and spatial reasoning. While recent multimodal large language models (MLLMs) have demonstrated impressive 2D image reasoning segmentation, adapting these capabilities to 3D scenes remains underexplored. In this paper, we introduce MLLM-For3D, a simple yet effective framework that transfers knowledge from 2D MLLMs to 3D scene understanding. Specifically, we utilize MLLMs to generate multi-view pseudo segmentation masks and corresponding text embeddings, then unproject 2D masks into 3D space and align them with the text embeddings. The primary challenge lies in the absence of 3D context and spatial consistency across multiple views, causing the model to hallucinate objects that do not exist and fail to target objects consistently. Training the 3D model with such irrelevant objects leads to performance degradation. To address this, we introduce a spatial consistency strategy to enforce that segmentation masks remain coherent in the 3D space, effectively capturing the geometry of the scene. Moreover, we develop a Token-for-Query approach for multimodal semantic alignment, enabling consistent identification of the same object across different views. Extensive evaluations on various challenging indoor scene benchmarks demonstrate that, even without any labeled 3D training data, MLLM-For3D outperforms existing 3D reasoning segmentation methods, effectively interpreting user intent, understanding 3D scenes, and reasoning about spatial relationships.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。