arXiv:2411.17772cs.CVcs.AI2024-11CVPR

用多视角生成提升3D重建质量,自建大规模训练数据集

MVBoost: Boost 3D Reconstruction with Multi-View Refinement

  • 通过多视角扩散模型生成图像,再用3D模型生成一致数据
  • 自建多视图数据集使模型重建精度显著提升
  • 适合需要高泛化能力的3D重建研究者使用

近期3D物体重建进展显著,但现有模型严重依赖已有3D数据集。由于多样化3D数据稀缺,导致重建模型泛化能力有限。本文提出一种多视角精炼增强3D重建的新框架(MVBoost),通过生成伪真值数据来突破瓶颈。核心思路是结合多视角生成模型的高精度与3D重建模型的一致性,构建可靠数据源:给定单张输入图像,先用多视角扩散模型生成多视图图像,再由大型3D重建模型生成一致性3D数据;随后对这些从3D数据渲染出的多视角图像进行自适应精炼,构建大规模多视图数据集,用于训练前馈式3D重建模型。此外,设计输入视图优化策略,根据用户输入图像自动优化对应视角,确保关键视点精准匹配用户需求。大量实验表明,该方法在重建效果和泛化能力上均优于现有方法。

原文摘要 · Abstract (English)

Recent advancements in 3D object reconstruction have been remarkable, yet most current 3D models rely heavily on existing 3D datasets. The scarcity of diverse 3D datasets results in limited generalization capabilities of 3D reconstruction models. In this paper, we propose a novel framework for boosting 3D reconstruction with multi-view refinement (MVBoost) by generating pseudo-GT data. The key of MVBoost is combining the advantages of the high accuracy of the multi-view generation model and the consistency of the 3D reconstruction model to create a reliable data source. Specifically, given a single-view input image, we employ a multi-view diffusion model to generate multiple views, followed by a large 3D reconstruction model to produce consistent 3D data. MVBoost then adaptively refines these multi-view images, rendered from the consistent 3D data, to build a large-scale multi-view dataset for training a feed-forward 3D reconstruction model. Additionally, the input view optimization is designed to optimize the corresponding viewpoints based on the user's input image, ensuring that the most important viewpoint is accurately tailored to the user's needs. Extensive evaluations demonstrate that our method achieves superior reconstruction results and robust generalization compared to prior works.

3D重建多视角扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。