将图像分割与3D形状对应,实现跨模态语义匹配。
Best Segmentation Buddies for Image-Shape Correspondence

- 用2D视觉模型特征蒸馏到3D表面,建立像素与顶点的相似性
- 通过'最佳分割伙伴'定位语义对应的3D顶点,精度高且鲁棒
- 适合做图像-3D形状对齐的科研人员和工程师
寻找对应关系是计算机视觉与图形学中的基础问题。本文研究了真实图像与无纹理3D形状之间的分割到分割对应任务,该任务因外观、几何和视角差异大而极具挑战。我们的方法通过将图像分割中的像素与3D形状对应语义部分的顶点关联,弥合跨模态差距。首先,从2D视觉模型中蒸馏深度特征至3D形状表面,计算图像像素与形状顶点间的特征相似性;其次,识别'最佳分割伙伴'——其最相似图像像素位于图像分割区域内,从而可靠地发现语义对应的3D顶点;最后,利用来自2D图像分割模型的蒸馏3D特征,直接在3D空间进行形状分割,推动对应过程自举。我们在大量图像-形状对上验证了方法的通用性与鲁棒性,实现了准确且语义合理的对应关系。
原文摘要 · Abstract (English)
Finding correspondences is a fundamental and extensively researched problem in computer vision and graphics. In this work, we examine the underexplored task of estimating segmentation-to-segmentation correspondence between images in the wild and untextured 3D shapes. This task is highly challenging due to substantial differences in appearance, geometry, and viewpoint. Our approach bridges the cross-modality gap by linking pixels in the image segment to vertices in the corresponding semantic part of the 3D shape. To achieve this, we first distill deep visual features from a 2D vision model onto the 3D shape surface, allowing for the computation of feature similarity between image pixels and shape vertices. Then, we identify Best Segmentation Buddies, vertices whose most similar image pixel lies within the image segmentation region, enabling the reliable discovery of vertices in semantically corresponding shape parts. Finally, we leverage distilled 3D features from the 2D image segmentation model to segment the shape directly in 3D, bootstrapping the correspondence process. We demonstrate the generality and robustness of our approach across a wide range of image-shape pairs, showcasing accurate and semantically meaningful correspondences. Our project page is at https://threedle.github.io/bsb/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。