仅用单目视频实现3D实例分割，无需预先标定数据
MoonSeg3R: Monocular Online Zero-Shot Segment Anything in 3D with Reconstructive Foundation Priors
- 利用重建基础模型从单张图像提取几何先验
- 在ScanNet200和SceneNN上达到与带深度数据方法相当的精度
- 适合实时3D场景理解，尤其适用于无深度传感器设备
本文聚焦于在线零样本单目3D实例分割这一新场景，现有方法因依赖带有姿态的RGB-D序列而失效。为此，我们采用近期提出的重建基础模型CUT3R，从单一RGB流中获取可靠的几何先验。提出MoonSeg3R，包含三个关键组件：(1) 基于空间语义蒸馏的自监督查询优化模块，将视觉基础模型生成的2D分割掩码转化为具有区分性的3D查询；(2) 3D查询索引记忆库，通过检索上下文查询实现时间一致性；(3) 来自CUT3R的状态分布令牌，作为掩码身份描述符增强跨帧融合。在ScanNet200和SceneNN上的实验表明，MoonSeg3R是首个实现在线单目3D分割的方法，性能媲美最先进的基于RGB-D系统。代码已开源。
原文摘要 · Abstract (English)
In this paper, we focus on online zero-shot monocular 3D instance segmentation, a novel practical setting where existing approaches fail to perform because they rely on posed RGB-D sequences. To overcome this limitation, we leverage CUT3R, a recent Reconstructive Foundation Model (RFM), to provide reliable geometric priors from a single RGB stream. We propose MoonSeg3R, which introduces three key components: (1) a self-supervised query refinement module with spatial-semantic distillation that transforms segmentation masks from 2D visual foundation models (VFMs) into discriminative 3D queries; (2) a 3D query index memory that provides temporal consistency by retrieving contextual queries; and (3) a state-distribution token from CUT3R that acts as a mask identity descriptor to strengthen cross-frame fusion. Experiments on ScanNet200 and SceneNN show that MoonSeg3R is the first method to enable online monocular 3D segmentation and achieves performance competitive with state-of-the-art RGB-D-based systems. Our code is available at https://github.com/VICO-UoE/MoonSeg3R.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。