通过图像特征反推3D形状与姿态,提升遮挡下检索精度。
PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Views

- 将检索转化为基于合成的特征分析,利用2D模型知识指导3D编码。
- 在完整与遮挡数据上均显著超越现有方法,尤其对局部特征敏感。
- 单框架实现形状检索、姿态估计与分类,适合真实场景应用。
单视图3D形状检索是日益重要的基础任务。现有方法多依赖对比学习或跨模态嵌入空间,但其全连接式对齐难以解释,泛化能力受限。为此,我们提出姿态感知3D形状检索(PASR),通过将DINOv3 2D基础模型的知识蒸馏至3D编码器,将检索建模为特征级分析-合成问题。通过对齐姿态相关的3D投影与2D特征图,PASR弥合了真实图像与合成网格之间的差距。推理时,PASR采用测试时优化,联合搜索最佳形状与姿态以重建输入图像的局部特征图。该合成优化机制对部分遮挡具有天然鲁棒性,并能捕捉细微几何细节。PASR在清洁与遮挡数据集上均大幅优于现有方法。此外,PASR具备强多任务能力,单一框架内实现鲁棒形状检索、竞争性姿态估计与准确类别分类。
原文摘要 · Abstract (English)
Single-view 3D shape retrieval is a fundamental yet challenging task that is increasingly important with the growth of available 3D data. Existing approaches largely fall into two categories: those using contrastive learning to map point cloud features into existing vision-language spaces and those that learn a common embedding space for 2D images and 3D shapes. However, these feed-forward, holistic alignments are often difficult to interpret, which in turn limits their robustness and generalization to real-world applications. To address this problem, we propose Pose-Aware 3D Shape Retrieval (PASR), a framework that formulates retrieval as a feature-level analysis-by-synthesis problem by distilling knowledge from a 2D foundation model (DINOv3) into a 3D encoder. By aligning pose-conditioned 3D projections with 2D feature maps, our method bridges the gap between real-world images and synthetic meshes. During inference, PASR performs a test-time optimization via analysis-by-synthesis, jointly searching for the shape and pose that best reconstruct the patch-level feature map of the input image. This synthesis-based optimization is inherently robust to partial occlusion and sensitive to fine-grained geometric details. PASR substantially outperforms existing methods on both clean and occluded 3D shape retrieval datasets by a wide margin. Additionally, PASR demonstrates strong multi-task capabilities, achieving robust shape retrieval, competitive pose estimation, and accurate category classification within a single framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。