用预训练扩散模型实现单图新视角生成,不需微调且细节清晰。
Novel View Synthesis using DDIM Inversion
- 通过相机位姿条件的U-Net直接预测目标视角的隐空间表示。
- 在MVImgNet上达到4.39的PSNR,优于现有方法。
- 利用DDIM反演噪声结构融合,保留纹理细节,适合快速生成场景视图。
从单张输入图像生成新视角是一项挑战性任务,需要推断场景的三维结构并恢复遮挡区域的细节,同时保持多视角间的几何一致性。现有方法通常需对大型扩散模型进行多视图微调或从头训练,成本极高,且存在重建模糊、泛化能力差等问题。为此,本文提出一种轻量级显式视角转换框架,直接利用预训练扩散模型的高质量生成能力,仅基于单图的DDIM反演隐空间表示,通过相机位姿条件的TUNet预测目标视角对应的隐空间表示。为解决预测隐空间采样后图像模糊的问题,提出一种新颖的融合策略,利用DDIM反演中观察到的固有噪声相关性结构,有效保留纹理与细粒度细节。最终使用融合后的隐空间作为初始条件进行DDIM采样,借助预训练扩散模型的生成先验完成新视角合成。在MVImgNet上的大量实验表明,本方法性能超越现有方法。
原文摘要 · Abstract (English)
Synthesizing novel views from a single input image is a challenging task. It requires extrapolating the 3D structure of a scene while inferring details in occluded regions, and maintaining geometric consistency across viewpoints. Many existing methods must fine-tune large diffusion backbones using multiple views or train a diffusion model from scratch, which is extremely expensive. Additionally, they suffer from blurry reconstruction and poor generalization. This gap presents the opportunity to explore an explicit lightweight view translation framework that can directly utilize the high-fidelity generative capabilities of a pretrained diffusion model while reconstructing a scene from a novel view. Given the DDIM-inverted latent of a single input image, we employ a camera pose-conditioned translation U-Net, TUNet, to predict the inverted latent corresponding to the desired target view. However, the image sampled using the predicted latent may result in a blurry reconstruction. To this end, we propose a novel fusion strategy that exploits the inherent noise correlation structure observed in DDIM inversion. The proposed fusion strategy helps preserve the texture and fine-grained details. To synthesize the novel view, we use the fused latent as the initial condition for DDIM sampling, leveraging the generative prior of the pretrained diffusion model. Extensive experiments on MVImgNet demonstrate that our method outperforms existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。