仅用2D图像即可快速分割任意3D物体,无需针对场景训练。
WildSeg3D: Segment Any 3D Objects in the Wild from 2D Images
- 基于前馈机制,通过动态对齐提升多视角3D点云精度。
- 相比顶尖模型提速40倍,且保持同等分割准确率。
- 适合实时交互场景,支持快速响应用户标注提示。
近期基于2D图像的交互式3D分割方法取得了显著进展,但现有模型通常需要大量场景特定训练才能准确重建与分割物体,限制了其在实时场景中的应用。本文提出WildSeg3D,一种高效方法,仅通过前馈机制即可实现跨多样化环境对任意3D物体的分割。前馈方法面临多视角间3D对齐误差累积的问题,导致分割结果失准。为此,我们提出动态全局对齐(DGA),通过动态调整函数聚焦难以匹配的3D点以改善全局多视角对齐精度。此外,为支持实时交互分割,我们引入多视角分组映射(MGM),利用物体掩码缓存整合多视角分割结果,快速响应用户输入。WildSeg3D在任意场景下均表现出强泛化能力,无需场景特定训练。实验表明,该方法不仅达到当前最优(SOTA)水平的分割精度,还相较现有SOTA模型实现40倍加速。代码将公开。
原文摘要 · Abstract (English)
Recent advances in interactive 3D segmentation from 2D images have demonstrated impressive performance. However, current models typically require extensive scene-specific training to accurately reconstruct and segment objects, which limits their applicability in real-time scenarios. In this paper, we introduce WildSeg3D, an efficient approach that enables the segmentation of arbitrary 3D objects across diverse environments using a feed-forward mechanism. A key challenge of this feed-forward approach lies in the accumulation of 3D alignment errors across multiple 2D views, which can lead to inaccurate 3D segmentation results. To address this issue, we propose Dynamic Global Aligning (DGA), a technique that improves the accuracy of global multi-view alignment by focusing on difficult-to-match 3D points across images, using a dynamic adjustment function. Additionally, for real-time interactive segmentation, we introduce Multi-view Group Mapping (MGM), a method that utilizes an object mask cache to integrate multi-view segmentations and respond rapidly to user prompts. WildSeg3D demonstrates robust generalization across arbitrary scenes, thereby eliminating the need for scene-specific training. Specifically, WildSeg3D not only attains the accuracy of state-of-the-art (SOTA) methods but also achieves a $40\times$ speedup compared to existing SOTA models. Our code will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。