用一张图精准还原物体3D姿态与形状,对遮挡和罕见视角仍有效。
SDFit: 3D Object Pose and Shape by Fitting a Morphable SDF to a Single Image
- 基于可变形符号距离函数,迭代优化姿态与形状
- 在三个数据集上表现优于现有方法,尤其抗遮挡能力强
- 无需重新训练,适合真实场景的泛化应用
从单张图像恢复3D物体姿态与形状是极具挑战性的病态问题,受严重遮挡、深度模糊、类内类间形状差异大及真实图像缺乏3D真值等因素影响。现有深度网络方法依赖合成数据训练,难以泛化到真实图像,且缺乏精炼噪声估计的反馈机制,主要关注几何而忽略像素对齐。为此,我们提出SDFit——一种新的渲染-比对优化框架。其三大创新:首先,采用学习得到的类别特定可变形符号距离函数(mSDF)模型,通过迭代优化同时调整3D姿态与形状,约束搜索空间于有效形状流形,支持任意拓扑;其次,利用基础模型高效检索匹配的初始3D形状;第三,通过基础特征建立图像与mSDF间的丰富2D-3D对应关系初始化姿态。在Pix3D、Pascal3D+和COMIC三个图像数据集上评估,SDFit在无遮挡和常见姿态下达到最先进水平,且对遮挡和非常见姿态具有独特鲁棒性,无需为未见图像重新训练。代码已公开于https://anticdimi.github.io/sdfit。
原文摘要 · Abstract (English)
Recovering 3D object pose and shape from a single image is a challenging and ill-posed problem. This is due to strong (self-)occlusions, depth ambiguities, the vast intra- and inter-class shape variance, and the lack of 3D ground truth for natural images. Existing deep-network methods are trained on synthetic datasets to predict 3D shapes, so they often struggle generalizing to real-world images. Moreover, they lack an explicit feedback loop for refining noisy estimates, and primarily focus on geometry without directly considering pixel alignment. To tackle these limitations, we develop a novel render-and-compare optimization framework, called SDFit. This has three key innovations: First, it uses a learned category-specific and morphable signed-distance-function (mSDF) model, and fits this to an image by iteratively refining both 3D pose and shape. The mSDF robustifies inference by constraining the search on the manifold of valid shapes, while allowing for arbitrary shape topologies. Second, SDFit retrieves an initial 3D shape that likely matches the image, by exploiting foundational models for efficient look-up into 3D shape databases. Third, SDFit initializes pose by establishing rich 2D-3D correspondences between the image and the mSDF through foundational features. We evaluate SDFit on three image datasets, i.e., Pix3D, Pascal3D+, and COMIC. SDFit performs on par with SotA feed-forward networks for unoccluded images and common poses, but is uniquely robust to occlusions and uncommon poses. Moreover, it requires no retraining for unseen images. Thus, SDFit contributes new insights for generalizing in the wild. Code is available at https://anticdimi.github.io/sdfit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。