LESS仅用二值掩码实现高效单阶段3D指代分割,性能超越此前方法3.7% mIoU。
LESS: Label-Efficient and Single-Stage Referring 3D Segmentation
- 单阶段设计,融合点与文本特征对齐,避免两阶段分离训练
- 仅需二值标签监督,在ScanRefer上达41.6% mIoU,比前人高3.7%
- 适合标注成本敏感、追求端到端效率的3D视觉-语言任务研究者
指代3D分割是一项视觉-语言任务,旨在从点云中分割出由句子描述指定的物体。以往方法采用两阶段范式:先进行语言无关的实例分割,再与文本查询匹配。但语义概念在训练中被分开展示,且需为每个对象提供实例和语义标签,耗时且人力密集。为此,我们提出新型单阶段、标签高效的分割框架LESS,仅依赖二值掩码监督。设计点-词跨模态对齐模块,实现点与文本嵌入的细粒度特征对齐;引入查询掩码预测模块与查询-句子对齐模块,完成粗粒度掩码与查询的匹配;提出区域正则化损失,大规模抑制无关背景预测;同时引入点对点对比损失,增强细微相似特征点的区分能力。大量实验表明,仅使用二值标签,LESS在ScanRefer数据集上达到41.6% mIoU,领先此前方法约3.7%。代码已开源。
原文摘要 · Abstract (English)
Referring 3D Segmentation is a visual-language task that segments all points of the specified object from a 3D point cloud described by a sentence of query. Previous works perform a two-stage paradigm, first conducting language-agnostic instance segmentation then matching with given text query. However, the semantic concepts from text query and visual cues are separately interacted during the training, and both instance and semantic labels for each object are required, which is time consuming and human-labor intensive. To mitigate these issues, we propose a novel Referring 3D Segmentation pipeline, Label-Efficient and Single-Stage, dubbed LESS, which is only under the supervision of efficient binary mask. Specifically, we design a Point-Word Cross-Modal Alignment module for aligning the fine-grained features of points and textual embedding. Query Mask Predictor module and Query-Sentence Alignment module are introduced for coarse-grained alignment between masks and query. Furthermore, we propose an area regularization loss, which coarsely reduces irrelevant background predictions on a large scale. Besides, a point-to-point contrastive loss is proposed concentrating on distinguishing points with subtly similar features. Through extensive experiments, we achieve state-of-the-art performance on ScanRefer dataset by surpassing the previous methods about 3.7% mIoU using only binary labels. Code is available at https://github.com/mellody11/LESS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。