无需训练，用大模型给3D体素分组并命名，实现开放词汇场景理解
OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding
- 直接用多模态大模型进行文本搜索，不依赖预训练文本编码器
- 在复杂指代分割任务中表现优于现有方法，尤其在开放词汇场景下
- 适用于需要理解任意物体名称的3D场景分析，如机器人导航
我们提出 OpenVoxel，一种无需训练的算法，用于对稀疏体素进行分组与标注，以实现开放词汇的3D场景理解。基于从多视角图像生成的稀疏体素栅格（SVR）模型，OpenVoxel 能够生成描述场景中不同物体的有意义分组。通过利用强大的视觉语言模型（VLMs）和多模态大语言模型（MLLMs），OpenVoxel 成功构建出信息丰富的场景地图，为每个分组生成语义描述，从而支持后续的开放词汇分割（OVS）或指代表达分割（RES）等任务。与以往方法不同，本方法无需训练，也不引入 CLIP/BERT 类文本编码器的嵌入表示，而是直接采用 MLLMs 进行文本到文本搜索。大量实验表明，该方法在近期研究中表现更优，尤其在复杂的指代表达分割任务中优势显著。代码将开源。
原文摘要 · Abstract (English)
We propose OpenVoxel, a training-free algorithm for grouping and captioning sparse voxels for the open-vocabulary 3D scene understanding tasks. Given the sparse voxel rasterization (SVR) model obtained from multi-view images of a 3D scene, our OpenVoxel is able to produce meaningful groups that describe different objects in the scene. Also, by leveraging powerful Vision Language Models (VLMs) and Multi-modal Large Language Models (MLLMs), our OpenVoxel successfully build an informative scene map by captioning each group, enabling further 3D scene understanding tasks such as open-vocabulary segmentation (OVS) or referring expression segmentation (RES). Unlike previous methods, our method is training-free and does not introduce embeddings from a CLIP/BERT text encoder. Instead, we directly proceed with text-to-text search using MLLMs. Through extensive experiments, our method demonstrates superior performance compared to recent studies, particularly in complex referring expression segmentation (RES) tasks. The code will be open.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。