arXiv:2601.07333cs.CVcs.RO2026-01被引 3

仅用一张图和一句话,就能从海量模型中找到最匹配的3D物体。

OSCAR: Open-Set CAD Retrieval from a Language Prompt and a Single Image

  • 通过图文双模态匹配,从无标注数据库中检索3D模型。
  • 在YCB-V数据集上检索准确率达90.48%,优于现有方法。
  • 适合需要快速适配新物体的机器人与增强现实场景。

6D物体位姿估计在机器人和增强现实等场景中至关重要。为应对部署后物体集合不断变化的需求,现代零样本位姿估计算法无需特定物体训练,仅依赖CAD模型。但实际部署中难以获取模型,且对象持续增长使精准识别目标实例更难。为此,我们提出OSCAR:一种无需训练的开放集CAD检索方法,可从无标注3D物体数据库中检索匹配模型。上线阶段,OSCAR生成数据库模型的多视角渲染图,并用图像描述模型生成描述性标题;推理时,GroundedSAM检测输入图像中的目标物体,分别计算感兴趣区域与数据库标题的多模态嵌入。OSCAR采用两阶段检索:先用CLIP进行文本过滤,再用DINOv2进行图像相似度精修。实验表明,OSCAR在跨域3D模型检索基准MI3DOR上超越所有现有方法。此外,我们验证了其在6D位姿估计中自动获取物体模型的应用价值:当无法获取精确实例时,使用最相似模型进行位姿估计,OSCAR在YCB-V数据集上达到90.48%的平均精度。进一步实验显示,该方法在Megapose中表现优于基于重建的方法。

原文摘要 · Abstract (English)

6D object pose estimation plays a crucial role in scene understanding for applications such as robotics and augmented reality. To support the needs of ever-changing object sets in such context, modern zero-shot object pose estimators were developed to not require object-specific training but only rely on CAD models. Such models are hard to obtain once deployed, and a continuously changing and growing set of objects makes it harder to reliably identify the instance model of interest. To address this challenge, we introduce an Open-Set CAD Retrieval from a Language Prompt and a Single Image (OSCAR), a novel training-free method that retrieves a matching object model from an unlabeled 3D object database. During onboarding, OSCAR generates multi-view renderings of database models and annotates them with descriptive captions using an image captioning model. At inference, GroundedSAM detects the queried object in the input image, and multi-modal embeddings are computed for both the Region-of-Interest and the database captions. OSCAR employs a two-stage retrieval: text-based filtering using CLIP identifies candidate models, followed by image-based refinement using DINOv2 to select the most visually similar object. In our experiments we demonstrate that OSCAR outperforms all state-of-the-art methods on the cross-domain 3D model retrieval benchmark MI3DOR. Furthermore, we demonstrate OSCAR's direct applicability in automating object model sourcing for 6D object pose estimation. We propose using the most similar object model for pose estimation if the exact instance is not available and show that OSCAR achieves an average precision of 90.48\% during object retrieval on the YCB-V object dataset. Moreover, we demonstrate that the most similar object model can be utilized for pose estimation using Megapose achieving better results than a reconstruction-based approach.

6D位姿估计3D检索零样本多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。