提出快速无提案的3D实例分割模型,速度提升百倍且精度超越现有方法。
SpaCeFormer: Fast Proposal-Free Open-Vocabulary 3D Instance Segmentation

- 基于空间窗口注意力与默顿曲线序列化,直接从查询预测实例掩码。
- 在ScanNet200上零样本mAP达11.1,较前人提升2.8倍。
- 适用于机器人与AR/VR场景,适合追求实时性与开放词汇能力的研究者。
开放词汇3D实例分割对机器人和增强现实/虚拟现实至关重要,但以往方法存在瓶颈:多阶段2D+3D流水线每场景耗时数百秒,而端到端伪标签方法依赖碎片化掩码和外部区域提议。我们提出SpaCeFormer,一种无提案的空间-曲线变换器,在标准基准上每场景仅需0.12–0.30秒,比多阶段方法快2–3个数量级。我们还构建了SpaCeFormer-3M,目前最大的开放词汇3D实例分割数据集(包含7.4K场景中604K实例的300万条多视图一致描述)。该数据集使掩码召回率相比单视图方法提高21倍(IoU>0.5时达54.3% vs 2.5%)。SpaCeFormer结合空间窗口注意力与默顿曲线序列化实现空间一致性特征表示,并采用RoPE增强解码器,直接从学习查询预测实例掩码,无需外部提议。在ScanNet200上达到11.1的零样本mAP,是此前最佳无提案方法的2.8倍;在ScanNet++和Replica上分别取得22.9和24.1 mAP,超越所有先前方法,包括使用多视图2D输入的方法。
原文摘要 · Abstract (English)
Open-vocabulary 3D instance segmentation is a core capability for robotics and AR/VR, but prior methods trade one bottleneck for another: multi-stage 2D+3D pipelines aggregate foundation-model outputs at hundreds of seconds per scene, while pseudo-labeled end-to-end approaches rely on fragmented masks and external region proposals. We present SpaCeFormer, a proposal-free space-curve transformer that runs in 0.12--0.30 seconds per scene across standard benchmarks, 2--3 orders of magnitude faster than multi-stage 2D+3D pipelines. We pair it with SpaCeFormer-3M, the largest open-vocabulary 3D instance segmentation dataset (3.0M multi-view-consistent captions over 604K instances from 7.4K scenes) built through multi-view mask clustering and multi-view VLM captioning; it reaches 21$\times$ higher mask recall than prior single-view pipelines (54.3% vs 2.5% at IoU$>$0.5). SpaCeFormer combines spatial window attention with Morton-curve serialization for spatially coherent features, and uses a RoPE-enhanced decoder to predict instance masks directly from learned queries without external proposals. On ScanNet200 we achieve 11.1 zero-shot mAP, a 2.8$\times$ improvement over the prior best proposal-free method; on ScanNet++ and Replica, we reach 22.9 and 24.1 mAP, surpassing all prior methods including those using multi-view 2D inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。