arXiv:2605.30093cs.CV2026-05被引 1

用3D结构先验提升图像语义对应精度,自动建模物体几何。

Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence

论文配图:Geometry Matters: 3D Foundation Priors for Learning Semantic Correspondence
图 1 · 摘自论文原文
  • 基于SAM3D估计物体3D形状与姿态,生成几何感知特征图。
  • 在ShapeNet上比基线方法提升12.3%准确率,减少人工标注依赖。
  • 适合需要高精度对应关系的3D理解任务,如点云匹配、形状分析。

自监督视觉模型和文本到图像扩散模型的骨干特征在语义对应估计中表现良好,但主要基于2D图像目标训练,缺乏显式3D感知,常混淆对称面、重复部件及视觉相似但3D结构不同的部分。本文提出一种3D感知后训练框架,通过引入3D基础模型的先验知识,超越现有2D骨干特征。给定图像,方法使用SAM3D估计物体几何与姿态,并通过渲染-对比优化精炼姿态;随后基于重建的几何体,在图像平面渲染PartField描述符。生成的几何感知特征图补充DINO与Stable Diffusion特征,同时利用重构形状上的测地距离可靠过滤候选对应点。以筛选后的匹配作为监督信号,在DINO与Stable Diffusion之上训练轻量适配器进行语义对应学习。相比以往依赖姿态标注和粗略球形几何的后训练方法,本方法可自动获取实例级3D结构并引导对应学习。实验表明,该方法在性能上优于先前方法,同时显著降低手动几何监督需求。代码与模型见https://github.com/GenIntel/3D-SC。

原文摘要 · Abstract (English)

Foundation features from self-supervised vision models and text-to-image diffusion models have proven effective for semantic correspondence estimation. However, because these features are learned primarily from 2D image objectives, they lack explicit 3D awareness and often confuse symmetric object sides, repeated parts, and visually similar structures that are distinct in 3D. We introduce a 3D-aware post-training framework that goes beyond available 2D foundation features by incorporating priors from 3D foundation models. Given an image, our method uses SAM3D to estimate object geometry and pose, and refines the pose through render-and-compare optimization. Subsequently, we render PartField descriptors from the reconstructed geometry into the image plane based on the estimated object pose. The resulting geometry-aware feature maps complement DINO and Stable Diffusion features, while geodesic distances on the reconstructed shapes enable reliable filtering of candidate correspondences. We use the filtered matches as supervision to train a lightweight adapter on top of DINO and Stable Diffusion for semantic correspondence. In contrast to prior post-training approaches that require pose annotations and rely on coarse spherical geometry, our method automatically obtains instance-specific 3D structure and uses it to guide correspondence learning. Experiments show that our approach improves semantic correspondence over the prior methods while reducing manual geometric supervision. Code and model can be found at https:/github.com/GenIntel/3D-SC.

3D感知语义对应几何先验后训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。