arXiv:2410.09467cs.CV2024-10被引 4

融合2D与3D扩散模型优势,提升单图生成3D的细节与一致性。

Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors

  • 用频域蒸馏损失结合高斯点云,融合2D/3D扩散模型优点。
  • 生成结果在几何一致性与纹理细节上均超越当前最优方法。
  • 适合需要高质量3D重建的AR/VR、机器人抓取等应用。

从一张自然拍摄的未对齐RGB图像生成3D物体,需估计其完整三维几何与纹理。准确重建物体的3D结构和纹理在机器人操作、抓取、3D场景理解及AR/VR中具有广泛应用。现有方法利用高斯点云优化,并由预训练的2D或3D扩散模型引导生成。然而,这些模型训练数据差异大,导致输出不一致:2D模型生成纹理细节丰富但跨视角不一致;3D模型保持视图间一致性却纹理过于平滑。本文提出通过两阶段频域蒸馏损失,将3D扩散模型的低频几何先验与2D扩散模型的高频纹理细节相结合,以高斯点云为载体,实现更精细且一致的3D重建。实验表明,该方法在几何一致性与视觉质量上均优于当前SOTA。此外,方法可高效用于物体姿态估计与跟踪。

原文摘要 · Abstract (English)

3D object generation from a single image involves estimating the full 3D geometry and texture of unseen views from an unposed RGB image captured in the wild. Accurately reconstructing an object's complete 3D structure and texture has numerous applications in real-world scenarios, including robotic manipulation, grasping, 3D scene understanding, and AR/VR. Recent advancements in 3D object generation have introduced techniques that reconstruct an object's 3D shape and texture by optimizing the efficient representation of Gaussian Splatting, guided by pre-trained 2D or 3D diffusion models. However, a notable disparity exists between the training datasets of these models, leading to distinct differences in their outputs. While 2D models generate highly detailed visuals, they lack cross-view consistency in geometry and texture. In contrast, 3D models ensure consistency across different views but often result in overly smooth textures. We propose bridging the gap between 2D and 3D diffusion models to address this limitation by integrating a two-stage frequency-based distillation loss with Gaussian Splatting. Specifically, we leverage geometric priors in the low-frequency spectrum from a 3D diffusion model to maintain consistent geometry and use a 2D diffusion model to refine the fidelity and texture in the high-frequency spectrum of the generated 3D structure, resulting in more detailed and fine-grained outcomes. Our approach enhances geometric consistency and visual quality, outperforming the current SOTA. Additionally, we demonstrate the easy adaptability of our method for efficient object pose estimation and tracking.

3D生成高斯点云扩散模型单图重建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。