用几何感知生成修复3D高斯点云的缺陷,提升新视角合成质量。
Leveling3D: Leveling Up 3D Reconstruction with Feed-Forward 3D Gaussian Splatting and Geometry-Aware Generation
- 引入几何对齐适配器,让扩散模型生成符合3D结构的补全内容。
- 在公开数据集上实现新视角合成与深度估计的当前最优性能。
- 适合需要高质量3D重建和生成的视觉任务研究者使用。
前向3D重建已革新3D视觉领域,为新视角合成等下游任务提供强大基线。现有方法尝试通过扩散模型修复渲染瑕疵,但缺乏几何一致性,在外推视角的缺失区域无法有效填补。本文提出Leveling3D,一种融合前向3D重建与几何一致生成的新范式,实现重建与生成的联合优化。设计轻量级几何感知调优适配器,将扩散模型内部知识与前向模型的几何先验对齐,使生成内容可准确修复由3D表示欠约束区域导致的外推视角伪影。为增强生成多样性,提出调色板过滤训练策略,并在测试时采用掩码精修以避免修复区域边界混乱。更重要的是,经改进的外推视图可作为前向3D高斯溅射(3DGS)输入,进一步提升3D重建效果。在多个公开数据集上,该方法在新视角合成与深度估计任务中达到当前最优表现。
原文摘要 · Abstract (English)
Feed-forward 3D reconstruction has revolutionized 3D vision, providing a powerful baseline for downstream tasks such as novel-view synthesis with 3D Gaussian Splatting. Previous works explore fixing the corrupted rendering results with a diffusion model. However, they lack geometric concern and fail at filling the missing area on the extrapolated view. In this work, we introduce Leveling3D, a novel pipeline that integrates feed-forward 3D reconstruction with geometrical-consistent generation to enable holistic simultaneous reconstruction and generation. We propose a geometry-aware leveling adapter, a lightweight technique that aligns internal knowledge in the diffusion model with the geometry prior from the feed-forward model. The leveling adapter enables generation on the artifact area of the extrapolated novel views caused by underconstrained regions of the 3D representation. Specifically, to learn a more diverse distributed generation, we introduce the palette filtering strategy for training, and a test-time masking refinement to prevent messy boundaries along the fixing regions. More importantly, the enhanced extrapolated novel views from Leveling3D could be used as the inputs for feed-forward 3DGS, leveling up the 3D reconstruction. We achieve SOTA performance on public datasets, including tasks such as novel-view synthesis and depth estimation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。