arXiv:2409.07452cs.CVcs.MM2024-09被引 39

用视频扩散模型生成高分辨率3D图像,实现多视角一致且细节丰富。

Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models

论文配图:Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models
图 1 · 摘自论文原文
  • 将单图转多视角视为3D感知的序列图像生成(轨道视频)
  • 通过3D感知视频修复器提升至高分辨率纹理细节
  • 适合需要高质量3D重建与多视角一致图像的研究者

尽管图像到3D生成已取得显著进展,现有方法在生成多视角一致且具有高分辨率纹理细节的图像方面仍存在困难,尤其在缺乏3D感知的2D扩散范式下。本文提出高分辨率图像到3D生成模型Hi3D,一种基于视频扩散的新范式,将单图转多视图重新定义为3D感知的序列图像生成(即轨道视频生成)。该方法利用视频扩散模型中隐含的时间一致性知识,有效推广至3D生成中的多视角几何一致性。技术上,Hi3D首先通过引入3D先验(相机位姿条件)增强预训练视频扩散模型,生成低分辨率纹理的多视图图像;再训练一个3D感知的视频到视频修复器,进一步提升多视图图像至高分辨率纹理细节。这些高分辨率多视图图像通过3D高斯泼溅生成新视角,并最终借助3D重建获得高保真网格。在新视角合成与单视图重建任务上的大量实验表明,Hi3D能生成具有优异多视角一致性和高度细节纹理的图像。源代码与数据见:https://github.com/yanghb22-fdu/Hi3D-Official。

原文摘要 · Abstract (English)

Despite having tremendous progress in image-to-3D generation, existing methods still struggle to produce multi-view consistent images with high-resolution textures in detail, especially in the paradigm of 2D diffusion that lacks 3D awareness. In this work, we present High-resolution Image-to-3D model (Hi3D), a new video diffusion based paradigm that redefines a single image to multi-view images as 3D-aware sequential image generation (i.e., orbital video generation). This methodology delves into the underlying temporal consistency knowledge in video diffusion model that generalizes well to geometry consistency across multiple views in 3D generation. Technically, Hi3D first empowers the pre-trained video diffusion model with 3D-aware prior (camera pose condition), yielding multi-view images with low-resolution texture details. A 3D-aware video-to-video refiner is learnt to further scale up the multi-view images with high-resolution texture details. Such high-resolution multi-view images are further augmented with novel views through 3D Gaussian Splatting, which are finally leveraged to obtain high-fidelity meshes via 3D reconstruction. Extensive experiments on both novel view synthesis and single view reconstruction demonstrate that our Hi3D manages to produce superior multi-view consistency images with highly-detailed textures. Source code and data are available at \url{https://github.com/yanghb22-fdu/Hi3D-Official}.

3D生成视频扩散高分辨率多视角一致

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。