arXiv:2606.10594cs.CV2026-06被引 2

3D视觉语言分割新方法,直接用细粒度信息提升边界精度

Segment and Select: Vision-Language Segmentation in 3D Scenarios

论文配图:Segment and Select: Vision-Language Segmentation in 3D Scenarios
图 1 · 摘自论文原文
  • 跳过粗粒度超点,直接生成精细掩码候选
  • 结合大模型与空间语义,选出最优掩码,提升分割质量
  • 在ScanNet和Matterport3D上分别领先8.3和5.3 mIoU

3D视觉语言分割旨在根据语言指令和视觉观测对3D场景中的目标物体进行分割。以往方法严重依赖粗粒度超点表示以降低计算复杂度,导致分割质量差、边界模糊。本文提出SEGA3D范式,直接操作细粒度视觉信息,摆脱超点依赖。首先,通过掩码候选生成器提供细粒度类别掩码候选,显著提升候选掩码质量。其次,利用大语言模型(LLM)基于语言描述和视觉特征生成语义与空间信息,结合视觉特征输入语义-空间选择器(SSS),筛选出排名靠前的掩码候选。最后,设计回环验证模块(LVM)从选定候选中生成最终分割掩码。SEGA3D在ScanRefer、ScanNet和Matterport3D基准上表现优异,尤其在ScanNet和Matterport3D上分别超越当前最优方法8.3 mIoU和5.3 mIoU。代码将在发表后公开。

原文摘要 · Abstract (English)

3D vision-language segmentation aims to segment target objects in 3D scenarios according to the linguistic instructions and visual observations. Prior art heavily relies on the coarse superpoint representation to reduce the computation complexity, which suffers from poor segmentation quality and messy object boundaries. In this paper, we propose the SEGment-And-select (SEGA3D) paradigm for 3D visionlanguage segmentation that directly operates on the fine-grained visual information and is free from the superpoint dependency. Specifically, we first leverage a mask candidate generator to provide fine-grained categorical mask candidates, substantially improving the quality of candidate masks over the superpoint counterparts. Then, a Large Language Model (LLM) is utilized to generate the semantic and spatial information based on the linguistic description and visual features. The LLM output and visual features are fed to the Semantic-Spatial Selector (SSS) to produce the top-ranking mask candidates. Eventually, the Loopback Verification Module (LVM) is designed to yield the segmentation mask from the selected candidate masks. Our SEGA3D attains competitive performance on ScanRefer, ScanNet and Matterport3D benchmarks. Notably, our SEGA3D surpasses the top-performing counterpart by 8.3 mIoU and 5.3 mIoU on ScanNet and Matterport3D, respectively. Codes will be available upon publication.

3D分割视觉语言大模型细粒度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。