用2D检测器引导3D场景中罕见物体的快速精准定位
Retrieving Objects from 3D Scenes with Box-Guided Open-Vocabulary Instance Segmentation
- 从RGB图像生成3D实例掩码，结合2D开放词汇检测能力
- 处理速度显著提升，对训练数据少的物体仍能准确识别
- 适合机器人、增强现实等需实时检索稀有物体的场景
从场景级点云中定位和检索物体是机器人与增强现实领域的重要挑战。该任务通常被建模为开放词汇3D实例分割。尽管现有方法表现优异，但高度依赖SAM和CLIP从伴随点云的图像中生成并分类3D实例掩码，导致计算开销大、推理慢，难以部署于真实环境。Open-YOLO 3D通过使用实时2D检测器对点云直接生成的类无关掩码进行分类，省去SAM与CLIP，大幅降低推理时间。然而其在训练数据稀少的物体类别上泛化能力差。本文提出一种新方法：利用2D开放词汇检测器从RGB图像生成3D实例掩码，继承其对新类别识别能力，同时保持高效分类，实现对开放文本查询下罕见实例的快速精准检索。代码将开源于https://github.com/ndkhanh360/BoxOVIS。
原文摘要 · Abstract (English)
Locating and retrieving objects from scene-level point clouds is a challenging problem with broad applications in robotics and augmented reality. This task is commonly formulated as open-vocabulary 3D instance segmentation. Although recent methods demonstrate strong performance, they depend heavily on SAM and CLIP to generate and classify 3D instance masks from images accompanying the point cloud, leading to substantial computational overhead and slow processing that limit their deployment in real-world settings. Open-YOLO 3D alleviates this issue by using a real-time 2D detector to classify class-agnostic masks produced directly from the point cloud by a pretrained 3D segmenter, eliminating the need for SAM and CLIP and significantly reducing inference time. However, Open-YOLO 3D often fails to generalize to object categories that appear infrequently in the 3D training data. In this paper, we propose a method that generates 3D instance masks for novel objects from RGB images guided by a 2D open-vocabulary detector. Our approach inherits the 2D detector's ability to recognize novel objects while maintaining efficient classification, enabling fast and accurate retrieval of rare instances from open-ended text queries. Our code will be made available at https://github.com/ndkhanh360/BoxOVIS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。