用修复与补全扩散模型,让少图生成3D细节更真实。
RI3D: Few-Shot Gaussian Splatting With Repair and Inpainting Diffusion Priors
- 分两步重建:先修复可见区域,再补全缺失区域。
- 在极稀疏输入下,纹理细节优于当前最佳方法。
- 适合低数据场景的高质量3D内容生成,如文物重建。
本文提出RI3D,一种基于3D高斯溅射(3DGS)的新方法,利用扩散模型从少量输入图像中重建高质量新视角。核心创新是将视图合成分为两个任务:重建可见区域和补全缺失区域,并引入两个个性化扩散模型分别应对。其中一个模型(‘修复’)以渲染图像为输入,预测高质量图像,作为伪真值约束优化过程;另一个模型(‘补全’)专注于未观测区域的细节生成。为有效整合,采用两阶段优化策略:第一阶段用修复模型重建可见区域,第二阶段用补全模型重构缺失区域,并通过进一步优化保证一致性。此外,我们提出一种新型高斯初始化方法,结合3D一致性和平滑深度与高度详细的相对深度,获取每张图像的深度信息。实验表明,通过分离任务并分别使用修复与补全模型,可在极稀疏输入下生成具有精细纹理的可见与缺失区域,显著超越现有先进方法,在多样场景中表现优异。
原文摘要 · Abstract (English)
In this paper, we propose RI3D, a novel 3DGS-based approach that harnesses the power of diffusion models to reconstruct high-quality novel views given a sparse set of input images. Our key contribution is separating the view synthesis process into two tasks of reconstructing visible regions and hallucinating missing regions, and introducing two personalized diffusion models, each tailored to one of these tasks. Specifically, one model ('repair') takes a rendered image as input and predicts the corresponding high-quality image, which in turn is used as a pseudo ground truth image to constrain the optimization. The other model ('inpainting') primarily focuses on hallucinating details in unobserved areas. To integrate these models effectively, we introduce a two-stage optimization strategy: the first stage reconstructs visible areas using the repair model, and the second stage reconstructs missing regions with the inpainting model while ensuring coherence through further optimization. Moreover, we augment the optimization with a novel Gaussian initialization method that obtains per-image depth by combining 3D-consistent and smooth depth with highly detailed relative depth. We demonstrate that by separating the process into two tasks and addressing them with the repair and inpainting models, we produce results with detailed textures in both visible and missing regions that outperform state-of-the-art approaches on a diverse set of scenes with extremely sparse inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。