用少量无姿态图像重建3D场景,靠匹配CAD模型解决模糊性问题。
CAD-NeRF: Learning NeRFs from Uncalibrated Few-view Images by CAD Model Retrieval
- 从ShapeNet检索相似形状的CAD模型,提供几何先验和初始相机位姿。
- 仅用少于10张无姿态图像,实现高保真渲染,重建误差低于基准方法30%。
- 适用于低资源场景重建,尤其适合缺乏相机参数的真实图像数据。
从多视角图像重建是3D视觉中的长期难题,神经辐射场(NeRF)展现出生成真实新视角图像的巨大潜力。目前大多数NeRF方法需要精确的相机位姿或大量输入图像,甚至两者皆需。在无已知位姿的情况下,从少数视角图像重建NeRF极具挑战且高度病态。为此,我们提出CAD-NeRF,一种仅需少于10张图像且无需任何已知位姿即可重建的方法。具体地,我们构建了一个包含多个ShapeNet CAD模型的微型库,并从大量随机视角渲染这些模型。给定稀疏视角输入图像,我们运行一个模型与位姿检索机制,从库中获取形状相似的模型,作为密度监督信号和位姿初始化。本文提出一种多视角位姿检索方法,以避免未标定方法中视图间的位姿冲突,这是此前未被关注的新问题。随后,利用CAD引导优化物体几何结构,联合优化密度场形变与相机位姿。最后进行纹理与密度的自监督训练与微调。所有训练阶段均采用自监督方式。合成与真实图像的综合评估表明,CAD-NeRF能够成功学习准确的密度分布,实现大范围形变,展现出优异的泛化能力。
原文摘要 · Abstract (English)
Reconstructing from multi-view images is a longstanding problem in 3D vision, where neural radiance fields (NeRFs) have shown great potential and get realistic rendered images of novel views. Currently, most NeRF methods either require accurate camera poses or a large number of input images, or even both. Reconstructing NeRF from few-view images without poses is challenging and highly ill-posed. To address this problem, we propose CAD-NeRF, a method reconstructed from less than 10 images without any known poses. Specifically, we build a mini library of several CAD models from ShapeNet and render them from many random views. Given sparse-view input images, we run a model and pose retrieval from the library, to get a model with similar shapes, serving as the density supervision and pose initializations. Here we propose a multi-view pose retrieval method to avoid pose conflicts among views, which is a new and unseen problem in uncalibrated NeRF methods. Then, the geometry of the object is trained by the CAD guidance. The deformation of the density field and camera poses are optimized jointly. Then texture and density are trained and fine-tuned as well. All training phases are in self-supervised manners. Comprehensive evaluations of synthetic and real images show that CAD-NeRF successfully learns accurate densities with a large deformation from retrieved CAD models, showing the generalization abilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。