用GAN提升NeRF图像分辨率,兼顾3D一致性与计算效率
SuperNeRF-GAN: A Universal 3D-Consistent Super-Resolution Framework for Efficient and Enhanced 3D-Aware Image Synthesis
- 基于预训练NeRF生成器,通过深度引导的超分模块实现3D一致的高清图像
- 在多个数据集上达到9.2%的PSNR提升,推理速度比基线快3.5倍
- 适用于各类NeRF合成任务,适合需要高效高质量3D图像的开发者
神经体积渲染技术(如NeRF)已革新3D感知图像合成,可从不同视角生成单个场景或物体的图像。然而,NeRF的高计算成本限制了其在高分辨率(HR)图像合成中的应用。现有方法多采用2D超分辨率,牺牲3D一致性;部分方法虽提出辐射场流形或两阶段生成以实现3D一致的高分辨率合成,但仅适用于特定任务,通用性受限。为此,我们提出SuperNeRF-GAN,一个通用的3D一致超分辨率框架。该框架可无缝集成于基于NeRF的3D感知图像合成方法中,同时提升生成图像分辨率、保持3D一致性并降低计算开销。具体而言,给定能生成NeRF表示(如tri-plane)的预训练生成器,首先通过体渲染获得低分辨率图像及对应深度图和法向图;随后,采用NeRF超分辨率模块学习生成高分辨率NeRF;接着提出一种新颖的深度引导渲染流程,包含三步:通过深度聚合构建边界校正的多深度图、法向引导的深度超分辨率、以及深度引导的NeRF渲染。实验表明,该方法在效率、3D一致性和图像质量上均优于现有方法。消融实验证实了各组件的有效性。
原文摘要 · Abstract (English)
Neural volume rendering techniques, such as NeRF, have revolutionized 3D-aware image synthesis by enabling the generation of images of a single scene or object from various camera poses. However, the high computational cost of NeRF presents challenges for synthesizing high-resolution (HR) images. Most existing methods address this issue by leveraging 2D super-resolution, which compromise 3D-consistency. Other methods propose radiance manifolds or two-stage generation to achieve 3D-consistent HR synthesis, yet they are limited to specific synthesis tasks, reducing their universality. To tackle these challenges, we propose SuperNeRF-GAN, a universal framework for 3D-consistent super-resolution. A key highlight of SuperNeRF-GAN is its seamless integration with NeRF-based 3D-aware image synthesis methods and it can simultaneously enhance the resolution of generated images while preserving 3D-consistency and reducing computational cost. Specifically, given a pre-trained generator capable of producing a NeRF representation such as tri-plane, we first perform volume rendering to obtain a low-resolution image with corresponding depth and normal map. Then, we employ a NeRF Super-Resolution module which learns a network to obtain a high-resolution NeRF. Next, we propose a novel Depth-Guided Rendering process which contains three simple yet effective steps, including the construction of a boundary-correct multi-depth map through depth aggregation, a normal-guided depth super-resolution and a depth-guided NeRF rendering. Experimental results demonstrate the superior efficiency, 3D-consistency, and quality of our approach. Additionally, ablation studies confirm the effectiveness of our proposed components.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。